PHP 7.4 Modx 3.0.1 resources > web doesn't show

Did you see the post above about editing the class keys in the modx_site_content table?

Yes, there are no articles items in class_keys column. We never really used Articles in production so that’s not a surprise.

I’m checking all possible solutions because we actually have no viable options if this does not get resolved.

Also keep in mind, I always sync the remote to the dev database before doing updates, so the two databases, except for a path item, are the same, which means, the cause of the failure in theory is not related to any database issues, since my dev version works fine, at least in regards to having a usable resource web tree.

I’m puzzling this one out, I have taken a short look at the javascript, the tree stuff, some others, but I believe what is happening is that the main page js constructor encounters something that is missing or off on the live site, and this makes it correctly not try to send the request to populate the web resource tree, but simply skip it.

The thing that of course stands out about the javascript is that it’s almost impossible to debug, thousands of lines of code without a single debugger running, which is… sad. And some libraries with tens of thousands of characters in one single line, which are totally impossible to even examine in any real way.

The only positive is that I know it has to be something actual since both sites are running the same code, or should be, and have basically the same database data, so the difference is something, but what that something is I can’t say.

I don’t believe the issue is related to the original error I posted, I believe that error is the result of the actual failure, wherever that may be located.

I further believe that the 301 I see when forcing it is the result of modx core receiving a forced request for null data, which then triggers a forced return to the home page url, which then creates the html characters that create that error.

If this were a php error, we’d be well on our way, but it’s not, I check the logs repeatedly, and there are no php errors, tons of deprecated code errors, but no actual warnings or errors from the modx code.

I use light ajax too, but LIGHT, because I know how hard it is to do actual debugging, and that ajax when used does have possible debuggers built in since it is radically difficult to actually resolve bugs when dealing with php triggered by ajax requests, so I anticipate that issue by first keeping the ajax to a minimum, and certainly NOT allowing one big blob of javascript to then create repeated endless ajax queries, which is what is going on with this manager page code.

I also want to note, the failure is happening BEFORE the web resource tree is being built, because the real failure is that something happens which makes the page javascript not even try to build the web resource html tree data at all, the request is never sent on the failing page, which I believe means this issue is not related to anything in the actual tree or tree generation.

It’s not receiving a null or empty tree, it’s not sending the ajax request at all in the first place to the tree builder logic in connectors. In other words, it’s likely that the javascript already knows something is off or missing, and thus doesn’t fire the tree build request via ajax to connectors. In other words, the thing that generates the ajax request to the tree builder in connectors is probably wrapped by an if not empty conditional test, and failing that test means the tree builder is never fired.

This suggests strongly that something that is supposed to be defined and not empty is empty or not defined.

Also, the updater is scripted, and uses CLI php modx updater, and the same script updates both versions, so there are no variations there either, and all that can be verified and checked.

I’m extremely careful with this stuff, but I cannot be careful enough to fix this type of situation. I remain hopeful it’s something totally obscure that nobody has thought to check for, and that has slipped all notice, because if it’s not, we’re in serious trouble right now.

Also, the initial menu for the left bar is constructed out of a request to constructors, and that completes successfully, so the constructors has to be the right path in my opinion no matter what or the first menu would not get written either.

Also worth noting is that I can use the + item to ‘create resource’, since that doesn’t depend on the web resource tree being loaded.

But any edits or whatever aren’t possible currently unless I manually edit it in the database table, which is difficult to put it mildly, and obviously out of reach of the office staff.

Updated, thank you for helping making the list more complete :+1: Bob sent me another list of extras that are tested and ready to go the other day which I’ll add asap.

While I can be forgiving of being frustrated when an update fails, you do seem to be adamant about lashing out at everyone and every line of code. Please do try to temper yourself and appreciate the people here that are trying to help you, for free, with minimal information and zero access or knowledge of your setup beyond what you’ve provided.

3.0 will not be pulled because something in your site ended is incompatible. It’s a major upgrade for a reason and there are thousands of sites running 3.0 just fine.

As you’re very clearly not experienced in debugging MODX manager issues (and yes, there is legacy in 3.0 and things that don’t make sense if you try to follow it line-by-line), and in the interest of getting the problem solved and accurately identifying core issues that need resolution if any, please email mark@modmore.com a sudo manager login and I’ll look into it for you. The premium MODX support this normally falls under costs €115/hr but I’ll offer you one hour free of charge, which I reckon will be enough to sort it out. (If the file manager doesn’t work either, please also include sFTP/ssh, though from your description it sounds like it is just the resource tree that isn’t loading.)

When exactly did MODX become a commercial project? I’d love my share.

3 Likes

I’m frustrated because of the number of errors and unhandled events I’m seeing, several of which should have been removed in the early alpha prerelease stage of development, in particular, php deprecation alerts.

I’ll probably take you up on the offer for help but I want to first make absolutely sure I’ve covered every difference possible.

As a long time developer my concern is very simple: I am facing a non debuggable massive blob of javascript, or very difficult to debug, which is scary to put it mildly.

The only item not being loaded is the resource web json tree data, though I have been able to track that down to where it must be failing, but since it’s javascript blobs interacting with javascript classes extending javascript classes tracking that down is difficult, but I am going to take a stab at it today to see if i can at least try to narrow down where the failed test is happening, which may show what is making it fail. Luckily I can do these alert debugger tests on the live site since it’s in the manager and users won’t see that.

When I tried to donate to modx a while back, I got almost no interest, and was told they were focusing on their cloud commercial side of things, though I did finally manage to force a few hundred dollars onto them, but my impression was the interest was not in user donations but commercial activity. Thus my conclusion. The original plan was to do a recurring donation, but back then, there was no way to do that, so I gave up on it. Since I do thousands of hours of my own FOSS projects a year, I know what it means to do free software projects, and in general try not to take them for granted when I use them.

When I’m talking about lack of debuggers, I’m not speaking out of ignorance on this stuff, my first step for almost everything I do is the debuggers, even with ajax stuff, which I only do commercially, that’s my primary concern because code that is not debugable is only marginal robust or usable. One of my tools I somewhat jokingly refer to as a giant debugger that has a side effect of creating the output users like and want.

I realize that in theory, I can with pain get a working familiarity with the flow of code in the manager javascript, with some exceptions due to their compressed js stuff, but my hope is maybe later today to get some debuggers into the manager js so I can find where it’s failing, which should then let me find why it’s failing. I’ll probably fail in this however, but it’s worth a try, I know the strings to grep for so it should be a matter of finding where the conditional that wraps the tree builder call is located.

In terms of quip, I’ve already filed issue before with it on github, and was totally ignored, as were other people with similar issues, so there’s no way I’m touching the stuff until I am assured that what I do will be used and mainlined as soon as it’s tested. But I don’t know the codebase, I’ve only fixed specific items that were broken before, but I found it fairly easy to fix, but with the 3.0 move, it may not be as easy anymore.

Technically I could do that for the modx deprecated php stuff, which is generally easy to fix, just tedious, but that should have been done as the early pre 3.0 dev process was happening, not now, post release. It seemed clear that the devs were clear about the issue since they split the deprecation alert logs into their own section (rather than fix them?), but that was overall a good decision anyway.

I already use most, almost all, of my free time on doing free software projects, and modx is I do for work, though I actually have always liked it more or less since it’s very developer friendly on the cms side of things, and the time/energy costs to move away would be absolutely astronomical. That is the grounds for major anxiety, since we don’t actually have the option to move, so it has to work, the costs and time would be beyond what anyone would or should accept, so we have to figure this one out.

Basically I’m going to keep looking for every possible cause that might be different on dev and live systems, though it’s very difficult since no errors are being tripped anywhere during the Dashboard page build, it’s simply not getting data it needed, not an error, just no data, then skipping the tree build because of that.

But modx stuff I do not do for fun, it’s work, unfortunately, I have the ability technically to do it, but I also have finite time/desire to do coding in this way, which is why we decided to rely on modx for our site, which overall has been a good decision.

One thing I’ve found with testing and big new version rollouts is that often the majority of people using it are doing fresh installs, and the older the install is, the more likely something unanticipated happens because the testing was not seeing that scenario.

I also by the way maintain some legacy liibraries in our site that also spit out the same deprecation warnings that 3.0 is doing, but that code is about 20 years old, and is updated now and then to get rid of most failure and alerts and warnings, and it’s not distributed, so I’m the only person who has to be concerned about it.

Technically I could do a new install on the same server in a subdomain, to fully remove cloudflare from the equation, though I have tested by disabling cloudflare caching to see if that changed anything. It didn’t.

I actually here have to blame myself, I’ve seen for decades now how difficult major version updates are, and how long it takes to stabilize them top to bottom, it’s minimum 1 year, after primary release, just look at KDE 4.x, 5,x, Gnome 3.x, and my own stuff, which took about 1 year or more to stabilize on my total rewrite, and about 2 before I wasn’t getting any more real issues on it. And about 1 solid month on even a small internal refactor before it’s stable, and that’s with 10s of thousands of active users.

The error in the request handling code points to unexpected html in the connector request, as others have mentioned in this thread. The JS isn’t at fault here. However, there’s a SDK build you can use if you want the non-minified code to satisfy your curiosity.

Please do take me up on my offer as it’ll save you days of barking up the wrong tree.

1 Like

I will, but am concerned that you failed to grasp that the failure to load web tree event on dashboard occurs WITHOUT an error. It is only once you try to force the load that the error occurs, probably because there is something missing somewhere.

The initial post was a red herring, there are no real errors in the standard left nav resource tree panel build, it’s only when you try to force it by clicking on the web item that an error generates, probably because an impossible situation has been requested.

It took me a while to realize this, because I had not realized that manager page load always preloads the web resources tree data, unless it has failed to do so.

No offense, but Mark is very familiar with the MODX core code, and you are not. If you continue to poke around, you may end up creating more problems than he can solve in an hour.

And please try to be a little more grateful that people are trying to help you for free, in spite of all the insults you’ve hurled at them after performing a major version update without a decent site backup and ignoring the warnings about extras that might not work in the new version.

The backup we do daily, but unfortunately I didn’t notice this issue until way too late. I appreciate what you are saying here. Note that my updating procedure is probably several steps more careful than most people who do updates, but I did make one critical mistake, which I’ve made before because I always forget to test quip, which was not testing quip after my dev system update.

The difficulties in debugging this situation are what they are. The manager is very difficult to debug.

Some of the things we may be forced to fix at our expense, I’ll talk to my client about it. That’s quip.

I never poke around in terms of changing things when I don’t understand the code, though your warning there is correct, but I don’t do that until I understand what the problem is, then I might do some test stuff. That would be engineering basics, if you poke around and change things in a codebase you don’t know, the results are most likely to make things worse than better, and less resolvable.

I don’t recall any link to the page that listed the known supported/unsupported extensions on the update to 3 page, which I read several times. Maybe I missed it, but I only found it when researching the failure. However, I do totally agree, I failed to test it locally because I forgot to, which was definitely a mistake on my part. The live site manager failure however was totally not predictable, in fact, I can’t actually remember if it was failing when I did the update, I think I would have noticed it when checking the update, but I may have missed it, not realizing that the web resources list always appears open on load of dashboard.

Corner case failures are very difficult to anticipate however, you can try to plan for everything, which I do, but at some point, I’m going to miss something somewhere, that’s just how it works. Usually when it’s my code, that’s fine, I fix it, and move on, and often with other code, I diagnose then fix it (which I’ve done for quip several times), then move on, but in this case, we’ll see if it can be resolved.

I do appreciate the free as in freedom and free as in cost component of this all, which is why I had tried to do recurring payments to modx some years back, but there was no option for that so I let that slide due to other life issues.

Just so others are aware, @lizardx did take me up on my offer and we’re working through it. It looks like it may be a .htaccess rewrite rule that’s causing the connector request to get redirected. Will report back on the debugging and solution once it’s confirmed fixed.

2 Likes

Thanks Mark for your kind offer to help out and supporting a frustrated MODXer.

1 Like

I was so close to solving this, it was literally only because I missed two lines of security mod rewrite rules that I didn’t figure it out myself.

Note that what tripped this is that the 3.0 manager is mimicking what bad web actors present to websites in terms of certain url patterns. I don’t run those rules on my dev site, so it never triggered, but I will probably add them to my dev site so I can test this and narrow it down a bit more so I can restore most of the security rules.

I’ll get in touch with someone about the quip stuff, we can hold off a week or two on quip, and after that, it starts to become more pressing, so we may go ahead and do that update, it will depend on how much work it takes to get it all running.

For the record, about 50% of quip works fine, user inputs work once you remove one bit of spam stuff, database entries work, but the manager control panel doesn’t work, so you can’t edit or add posts, or delete them, and the emailer doesn’t work. So basically once you disable the two preview and create spam filters it works in terms of getting user data inputs, the recaptchav2 works fine, and sends its data to quip fine, so it’s I suspect only a small bit that requires some modification, I’d have to check into the code to confirm that…

In terms of the larger picture, this really brought home to me how non-trivial our binding to modx is a core dependency, which is something I have never been comfortable with, but unfortunately, modx has basically in my opinion an ideal developer framework with its templates, chunks, and snippets, and how they interact, along with wayfinder menus.

I will take this as a reminder however to always maintain the polilcy of always writing my own extensions, which is my practice, except for the barebones set we use.

The cause for my freakout here was realizing I had no control at all over the most essential element of our entire web presence, and that worse, there was no plan b on failure, which is not ideal for robust solutions.

Thanks mark, he spotted the questionable thing which I’d already tested, but realized there was a second rule, which I’d simply forgotten about, because I’d separated the two in the rewrite section, which is a mistake I have now corrected.

I’ll have to add a few steps required to my updating procedures, and sync my local and dev rewrite rules a bit more to catch more possible cases like this.

I know and hear what you are saying about free software communities, and the nature of the work and support, because I’ve done it myself for over 20 years, extensively, but there’s certain things I now realize that will trigger panic in me, and that is when I have no plan b to fall back on as the primary developer who is responsible for everything in this companies existence when it comes to the web. Maybe I take that too seriously, I don’t know, maybe not.

thanks

Note that since what we run is fairly standard security stuff for protection against malicious actors, and the actual thing is that modx 3 manager emulated one of those actions, I will now figure out the precise rule that trips the issue, and post it, so you can add it to your update page. Once I have that debugged I’ll look into the other issue, quip. Technically, what should probably happen once I determine the specific thing being done / sent by the manager is that it should be removed from that url modx sends so that such a change doesn’t need to be made in security rules, it’s certainly not necessary to how modx runs, it’s just something that happened without realizing I suspect.

For future reference, this rule:

RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|\bnull\b|%0A|%0D|%af|%a6|%bd|%bf|%c2|%c3|%e2).* [NC]
RewriteRule ^(.*)$ / [QSD,R=301,L]

was causing the connector request that loads the resources:

connectors/index.php?action=Resource/GetNodes&id=web_0&type=MODX\Revolution\modContext

to get redirected because MODX\Revolution is matching \\r in that regex, which then causing a html response where JSON is expected and ExtJS breaks.

@lizardx There’s nothing in MODX that’s wrong, here. MODX is not “emulating a malicious actor”. It’s triggering because your regex, which supposedly is meant to catch literal \r returns, is matching for anything that includes the string \r. The rule is flawed, period.

You can confirm that by restoring the rule, and trying a request like ?a=bla\r - it’ll redirect to the homepage.

I’ve never seen a snippet like that before and as MODX includes request sanitization I do doubt it’s necessary.

ah. I see it now, it’s those namespace urls, those were never seen in any standard url before, that’s why I didn’t find it when searching.

I don’t think it’s ‘flawed’ per se, it just failed to take into account that a path would ever use what appears to be escaped characters, like \r.

This is more a case of new data type which is not expected forcing a change in behavior.

Note that in traditional unix paths (aka, urls), you would NEVER see an escaped character, which is why looking for escaped characters was a totally legitimate thing to do. And still is.

However, I see the real solution in this case, it’s to leave the rule, which is absolutely correct, and add a rewrite condition to exclude manager or connectors.

I’ll give you the code later, but in general, I find that any solution that requires removing solid security stuff is not a solution, but one that adds in support for a new situation while retaining the old solution is a pretty good fix.

Note that modx doesn’t run our site, modx is our cms, period. But not all the content, just some of it. Our logic runs our site, and then when required, that is passed to modx depending on the request. Modx is involved in some capacities in almost all requests, but only as a side thing. I use absolute minimum features of modx to meet our needs, which works very well. It’s actually why I first picked evolution > revolution way back when, that core subset of modx was well designed, still is I think.

I suspect that someone who doesn’t use unix or linux may not realize how odd it is to pass an escaped character in a query string, or anything resembling that, but but most long time nix users would see that right away as a strange thing, wouldn’t have happened if I’d done it, for example, I probably would have translated the path/namespace thing to something else, and then translated it back in connectors or whatever.

There’s nothing solid about that “security fix”.

I told you exactly why it triggered and why it’s flawed when I pointed the rule out to you via email. Wont repeat myself again.

Your problem is solved, this topic is now locked.

With respect to rewriting quip, the MODX blog has a series of articles about rewriting extras for MODX 3. You might find it helpful. The first article is here.

The one on handling mail will be published soon.