New Extra: ExtraBuilder? Beta version to test

Hey All,

I uploaded a “quick” video… turned out to be 30 minutes :slight_smile:. It’s hard to know what the right amount of information is.

I’ve been working on a new Extra that makes it potentially quicker and easier to get started with building Extras. This is a graphical, table-driven Extra built into the manager that allows you to manage and define multiple Packages with a corresponding schema/tables and relationships. It also has a fairly simple Transport Package builder to allow you to package it up.

https://drive.google.com/file/d/1V8LWU4ZzeWW7HRuIRX9PsRgkfFuYbH-U/view?usp=sharing

** Edit: Download link updated 2021-01-17 07:00 AM MT to correct the namespace path

I used code and inspiration from multiple sources, but most of the tables.php resolver script is directly from the modExtra github project from Vasily (GitHub - modx-pro/modExtra: A base MODX Extra template to begin development on).

As others mentioned there is a very robust framework Git-Package-Manager that can do a lot more and is likely more flexible. As well, Bobray has the MyComponent extra which can get you going as well. Some other discussion around this here: Any interest? Extra to manage packages and custom tables - #11

So how is this different? What’s the goal?
Here’s my thoughts:

  1. This provides a shorter learning curve to create a package. You can setup a few custom tables, a snippet or two and some chunks and package it up in a very short amount of time.
  2. I think it’s a more intuitive starting point for new MODX users and those dabbling in development.
  3. A tool in the manager to allow more rapid prototyping of ideas. It feels like there is less to setup to test something out.

30 minute demo:

Hopefully the community finds it useful. For now I’m just linking it here. I didn’t want to post it to the Extra directory yet. If a few people are willing to try it out and report results that would be much appreciated! Feel free to log any issues against the repo:

PS: If anyone is interested in the mechanisms behind the scenes on how I am using Vuejs (Could be any JS framework) and an iFrame to create what feels like a “Native” MODX experience, let me know. I’m planning on putting together a couple more videos.

6 Likes

after installation the namespace core_path was /home/cabox/workspace/core/components/extrabuilder/
instead of {core_path}/components/extrabuilder/

changing that, makes the CMP running

Thanks for testing and catching that. I guess I need to add a different Modx environment with a different Core Path to the test plan :slight_smile:

I’ll get that corrected and reply back.

Package is corrected. I edited the original post download link to this one:

Hi, thanks for sharing the package, nice work. I tested the 1.0.1-beta. Namespace core-path setting is still missing a trailing slash. The getcategories processor is limited to a result of 20 items. The limit can be bypassed by adding a beforeQuery function.

public function beforeQuery() {
	$this->setProperty('limit', 0);
	return true;
}

Hey Raffy,

Thanks for the feedback! Adding the paging in is on my feature list :slight_smile:. I wanted to make sure it would be useful before spending more time on it.

Although, to your point, the amount of data is not likely to exceed a value that would cause slowness. So, I could probably do what you’re recommending and just return all results. I’ll do that to start with.

Also, do you think newer people to Modx would find this in the listings? I’m wondering if I might be better off calling this “TableCreator” or something referencing tables which might be a more likely search term.

Hey Jaredfhealy,

ExtraBuilder is ok for me, TableCreator seems a bit underrated. In addition to the function of creating tables, it would be useful to have the reverse way, creating a schema from existing tables .)

Ok, should have the namespace properly corrected this time. Checked the vehicle file generated for the namespace:

Added the package to Github so I can just link to those rather than google drive.

https://github.com/jaredfhealy/extrabuilder/raw/main/_packages/extrabuilder-1.0.2-beta.transport.zip

I’ll add in the idea to allow creating from an existing schema. I know there is a script out there to reverse engineer from the tables. I’ll take a look at how that works. I can probably just utilize that code and modify as needed.

@nadakbar Not sure if you saw this post. You expressed a little interest in a visual builder. If you have some time it would be great to get your feedback.

If not, no problem. Just trying to work out any big issues before putting it in the MODX Extras directory.

<?php
/*------------------------------------------------------------------------------
================================================================================
=== Reverse Engineer Existing MySQL Database Tables to xPDO Maps and Classes ===
================================================================================
 
SYNOPSIS:
This script generates the XML schema and PHP class files that describe custom
database tables.
 
This script is meant to be executed once only: after the class and schema files
have been created, the purpose of this script has been served.
 
USAGE:
1. Upload this file to the root of your MODx installation
2. Set the configuration details below
3. Navigate to this script in a browser to execute it,
    e.g. http://yoursite.com/thisscript.php
    or, you can do this via the command line, e.g. php this-script.php
 
INPUT:
Please configure the options below.
 
OUTPUT:
Creates XML and PHP files:
    core/components/$package_name/model/$package_name/*.class.php
    core/components/$package_name/model/$package_name/mysql/*.class.php
    core/components/$package_name/model/$package_name/mysql/*.inc.php
    core/components/$package_name/schema/$package_name.mysql.schema.xml
 
SEE ALSO:
http://modxcms.com/forums/index.php?topic=40174.0
http://rtfm.modx.com/display/revolution20/Using+Custom+Database+Tables+in+your+3rd+Party+Components
http://rtfm.modx.com/display/xPDO20/xPDOGenerator.writeSchema
------------------------------------------------------------------------------*/
 
/*------------------------------------------------------------------------------
        CONFIGURATION
------------------------------------------------------------------------------
Be sure to create a valid database user with permissions to the appropriate
databases and tables before you try to run this script, e.g. by running
something like the following:
 
CREATE USER 'your_user'@'localhost' IDENTIFIED BY 'y0urP@$$w0rd';
GRANT ALL ON your_db.* TO 'your_user'@'localhost';
FLUSH PRIVILEGES;
 
Be sure to test that the login criteria you created actually work before
continuing. If you *can* log in, but you receive errors (e.g. SQLSTATE[42000] [1044] )
when this script runs, then you may need to grant permissions for CREATE TEMPORARY TABLES
------------------------------------------------------------------------------*/
$debug = true;     // if true, will include verbose debugging info, including SQL errors.
$verbose = true;    // if true, will print status info.
 
// The XML schema file *must* be updated each time the database is modified, either
// manually or via this script. By default, the schema is regenerated.
// If you have spent time adding in composite/aggregate relationships to your
// XML schema file (i.e. foreign key relationships), then you may want to set this
// to 'false' in order to preserve your custom modifications.
$regenerate_schema = TRUE;
 
// Class files are not overwritten by default
$regenerate_classes = TRUE;
 
// Your package shortname:
$package_name = 'xxx';
 
 
// Database Login Info can be set explicitly:
$database_server = 'xxx';
$database_user = 'xxx';
$database_password = 'xxx';
$dbase = 'xxx';



// If your tables use a prefix, this will help identify them and it ensures that
// the class names appear "clean", without the prefix.
$table_prefix = 'modx_xxx';
// If you specify a table prefix, you probably want this set to 'true'. E.g. if you
// have custom tables alongside the modx_xxx tables, restricting the prefix ensures
// that you only generate classes/maps for the tables identified by the $table_prefix.
$restrict_prefix = true;
// OR, use your MODx Revo connection details.  Just uncomment the next line:
//include('core/config/config.inc.php');
 
 
 
 
//------------------------------------------------------------------------------
//  DO NOT TOUCH BELOW THIS LINE
//------------------------------------------------------------------------------
$base_path = realpath(dirname(__FILE__));
$xpdo_path = strtr( $base_path . '/core/xpdo/xpdo.class.php', '\\', '/');
include_once ( $xpdo_path );
 
// A few definitions of files/folders:
$package_dir = "$base_path/core/components/$package_name/";
$model_dir = "$base_path/core/components/$package_name/model/";
$class_dir = "$base_path/core/components/$package_name/model/$package_name";
$schema_dir = "$base_path/core/components/$package_name/model/schema";
$mysql_class_dir = "$base_path/core/components/$package_name/model/$package_name/mysql";
$xml_schema_file = "$base_path/core/components/$package_name/model/schema/$package_name.mysql.schema.xml";
 
// A few variables used to track execution times.
$mtime= microtime();
$mtime= explode(' ', $mtime);
$mtime= $mtime[1] + $mtime[0];
$tstart= $mtime;
 
// Validations
if ( empty($package_name) )
{
    print_msg('<h1>Reverse Engineering Error</h1>
        <p>The $package_name cannot be empty!  Please adjust the configuration and try again.</p>');
    exit;
}
 
// Create directories if necessary
$dirs = array($package_dir, $schema_dir ,$mysql_class_dir, $class_dir);
 
foreach ($dirs as $d)
{
    if ( !file_exists($d) )
    {
        if ( !mkdir($d, 0777, true) )
        {
            print_msg( sprintf('<h1>Reverse Engineering Error</h1>
                <p>Error creating <code>%s</code></p>
                <p>Create the directory (and its parents) and try again.</p>'
                , $d
            ));
            exit;
        }
    }
    if ( !is_writable($d) )
    {
        print_msg( sprintf('<h1>Reverse Engineering Error</h1>
            <p>The <code>%s</code> directory is not writable by PHP.</p>
            <p>Adjust the permissions and try again.</p>'
        , $d));
        exit;
    }
}
 
if ( $verbose )
{
    print_msg( sprintf('<br/><strong>Ok:</strong> The necessary directories exist and have the correct permissions inside of <br/>
        <code>%s</code>', $package_dir));
}
 
// Delete/regenerate map files?
if ( file_exists($xml_schema_file) && !$regenerate_schema && $verbose)
{
    print_msg( sprintf('<br/><strong>Ok:</strong> Using existing XML schema file:<br/><code>%s</code>',$xml_schema_file));
}
 
$xpdo = new xPDO("mysql:host=$database_server;dbname=$dbase", $database_user, $database_password, $table_prefix);
 
// Set the package name and root path of that package
$xpdo->setPackage($package_name, $package_dir, $package_dir);
$xpdo->setDebug($debug);
 
$manager = $xpdo->getManager();
$generator = $manager->getGenerator();
 
//Use this to create an XML schema from an existing database
if ($regenerate_schema)
{
    $xml = $generator->writeSchema($xml_schema_file, $package_name, 'xPDOObject', $table_prefix, $restrict_prefix);
    if ($verbose)
    {
        print_msg( sprintf('<br/><strong>Ok:</strong> XML schema file generated: <code>%s</code>',$xml_schema_file));
    }
}
 
// Use this to generate classes and maps from your schema
if ($regenerate_classes)
{
 
	    print_msg('<br/>Attempting to remove/regenerate class files...');
    delete_class_files( $class_dir );
    delete_class_files( $mysql_class_dir );
}
 
// This is harmless in and of itself: files won't be overwritten if they exist.
$generator->parseSchema($xml_schema_file, $model_dir);
 
$mtime= microtime();
$mtime= explode(" ", $mtime);
$mtime= $mtime[1] + $mtime[0];
$tend= $mtime;
$totalTime= ($tend - $tstart);
$totalTime= sprintf("%2.4f s", $totalTime);
 
if ($verbose)
{
    print_msg("<br/><br/><strong>Finished!</strong> Execution time: {$totalTime}<br/>");
 
    if ($regenerate_schema)
    {
        print_msg("<br/>If you need to define aggregate/composite relationships in your XML schema file, be sure to regenerate your class files.");
    }
}
 
exit ();
 
 
/*------------------------------------------------------------------------------
INPUT: $dir: a directory containing class files you wish to delete.
------------------------------------------------------------------------------*/
function delete_class_files($dir)
{
    global $verbose;
 
    $all_files = scandir($dir);
    foreach ( $all_files as $f )
    {
        if ( preg_match('#\.class\.php$#i', $f) || preg_match('#\.map\.inc\.php$#i', $f))
        {
            if ( unlink("$dir/$f") )
            {
                if ($verbose)
                {
                    print_msg( sprintf('<br/>Deleted file: <code>%s/%s</code>',$dir,$f) );
                }
            }
            else
            {
                print_msg( sprintf('<br/>Failed to delete file: <code>%s/%s</code>',$dir,$f) );
            }
        }
    }
}
/*------------------------------------------------------------------------------
Formats/prints messages.  The behavior is different if the script is run
via the command line (cli).
------------------------------------------------------------------------------*/
function print_msg($msg)
{
    if ( php_sapi_name() == 'cli' )
    {
        $msg = preg_replace('#<br\s*/>#i', "\n", $msg);
        $msg = preg_replace('#<h1>#i', '== ', $msg);
        $msg = preg_replace('#</h1>#i', ' ==', $msg);
        $msg = preg_replace('#<h2>#i', '=== ', $msg);
        $msg = preg_replace('#</h2>#i', ' ===', $msg);
        $msg = strip_tags($msg) . "\n";
    }
    print $msg;
}
 
/* EOF */
1 Like

Thanks for reaching out, just had a look through the youtube preview and it looks pretty solid from what i could see.

Currently mid project and using migx to do the same thing so it would be a good test once i have finished to see how well my current schema imports into the one you have created.

from what i have seen it looks like it is going to be a solid edition. Especially like the quick field types selector for common types such as text, int etc.

will update once i have a look but no idea when that may be.

No problem… Thanks for taking a look at the demo, I appreciate it!

I like the use of Vue.js and the navigation inside your extra. Makes the rest of the manager feel quite sluggish. Also the “Quick Select” for new fields is quite handy.

However, in my opinion the creation of a new relationship between tables could be improved.
It seems tedious to have to enter the same relationship for both objects individually. One form that creates both entries at the same time would help. And maybe dropdowns for available objects (and the available fields of these ojects) could avoid typos.

Also, there seems to be a conflation of aggregate/composite and the cardinality, that are not the same.
A “One to Many” relationship doesn’t have to be a composite relationships.

<object class="modUser" table="users" extends="modPrincipal">
	...
	<aggregate alias="CreatedResources" class="modResource" local="id" foreign="createdby" cardinality="many" owner="local" />
	<composite alias="Profile" class="modUserProfile" local="id" foreign="internalKey" cardinality="one" owner="local" />
</object>

For example a user can have many resources that he created, but when you delete the user, you don’t want to delete the resources. And a one-to-one-relationship (like the one between a user and the user-profile) can also be a composite relationships. The profile should be deleted when the user is deleted.


It seems that the JS part of your extra doesn’t take the setting of MODX_ASSETS_URL into account. My Modx installation is in a subfolder and I had to change the values of iframe.src in templates/home.html and transport.html and all the ajax-urls in appconfig.js and transport.js to make it work.

Great feedback. I definitely agree on the relationship aspect. I had thought about the same, that adding one side of the relationship could automatically generate the other, especially if that is the most common. I was seeing examples though where there were aggregate relationships to modUser without extending modUser (Specifically the doodles example). I think the effect would be I can call [myObject]->getOne('User'), but from the user object I would not be able to call [userObj]->getMany('MyObjects'). And deletion of my object would leave the related user object untouched.

I have to admit that I’m still a little fuzzy on relationships. I couldn’t find a reference that described them functionally very well. My understanding of the main types. Also, there doesn’t seem to be terminology to describe the xPDO relationships, unless I just missed it.

I’ll have to spend a little more time to fully understand the relationship variations and behavior. Until I understand it better, the interface will likely be limited by my understanding :slight_smile:.

If you have any references beyond the docs on the Modx site, let me know.

Maybe defining it something like this. Also is “Cascade delete” a common term? I’ve seen it used some places in describing the behavior where the related object is deleted. I’m trying to present the terminology in a way that is easier to understand from a functional perspective.

Two-Sided Relationships

  1. One to Many (With Cascade Delete, ex: Delete One and the Many are also deleted)
  2. One to One (With Cascade Delete, ex: Delete One and the related One is also deleted)

One-Sided Relationships

  1. One to Many (Without Cascade Delete, ex: Delete One and the Many remain)
  2. One to One (Without Cascade Delete, ex: Delete One and the related One remains)

The concept of Many to Many with a separate relationship table is probably out of scope for this first version.


I'll definitely review the iFrame aspect and make sure I set it to use dynamic paths and the Global path variables. I should be able to get that corrected this weekend and post back.

@halftrainedharry This should have the correction of the iFrame src path.
@raffenberg This version adds the reverse engineering although it works slightly differently. You can see a few details in the github issue:

https://github.com/jaredfhealy/extrabuilder/blob/3ec5e3ef76eb321bf8dc491d72e333960ca90e19/_packages/extrabuilder-1.0.3-beta.transport.zip

1 Like

I tried to install the new version and still had some issues.

I think this line


should be like this. (MODX_ASSETS_URL instead of MODX_CORE_PATH and = instead of .=).
$html = str_replace('${iframe_src_path}', MODX_ASSETS_URL.'components/extrabuilder/', $html);

The same is true for this line where MODX_ASSETS_PATH is used instead of MODX_ASSETS_URL and the concatenation should also be removed.

Additionally, the AJAX request in the files appconfig.js and transport.js still assume the standard assets path.

$.ajax({
	url: '/assets/components/extrabuilder/connector.php',
	...

Thank you for the persistence and thorough testing.

I added a couple steps to my testing process this time around and installed the Extra on a production server with a different directory structure. I also tested it on a Subdirectory Modx install. I think I have all the issues you mentioned fixed. I also corrected a couple other things I found as well as cleaning up the iFrame methodology a little bit to make it more “generic” and reusable.

I haven’t had time to work on the relationship interface or usage. I’m hoping to get this out in the Extra listings soon. Do you think the less than ideal relationship creation should be changed for an initial release or could this current version be an MVP to get out there?

https://github.com/jaredfhealy/extrabuilder/raw/main/_packages/extrabuilder-1.0.4-beta.transport.zip

PS: If anyone is interested in the mechanisms behind the scenes on how I am using Vuejs (Could be any JS framework) and an iFrame to create what feels like a “Native” MODX experience, let me know.
I’m planning on putting together a couple more videos.

Would love to see this if you get a chance sometime!

1 Like

I installed the new version and the installation works fine now.

However, there is probably a bug in the new functionality “Add Existing Tables to Package”.
I created a new table in the database with a column id (primary key, auto increment) and when I added the table to the package and looked at the schema, it looked like this:

<object class="Test" table="test" extends="xPDOSimpleObject">
	<field key="id" dbtype="int" precision="10" phptype="integer" null="false" default=""/>
	...
</object>

As the class “xPDOSimpleObject” already has a built-in, autoincrement field named id, there shouldn’t be another field id.

It also seems, that the created object always extends “xPDOSimpleObject”, even if the table has a primary key with a different name (or no primary key at all). Maybe it is a better idea to default to extend “xPDOObject” (and only change it to “xPDOSimpleObject” if there actually is a field id in the table).

1 Like

I’ll see if I can get at least a short walk through done this weekend.

1 Like