Importing 3rd party content. Where to Start?

Development
21

Have you seen ImportX extra, that might be helpful.

22

Okay so here’s where I’m at and I can’t figure out how to proceed.

My CSV file is just one a one column list now. I figured that I wouldn’t need the two preceding empty ones.

I have this code to run, which reads the CSV file, imports the data and a hashed value of it, reopens the CSV file and clears the file.

$base_path = $modx->getOption('core_path').'components/myPackage/';
$modx->addPackage('myPackage', $base_path.'model/');


$csvPath = "./path/to/list/export-schools.csv";
$CSVfp = fopen( $csvPath, "r" );
if ( $CSVfp !== FALSE ) :
	while ( !feof( $CSVfp ) ) :
		$data = fgetcsv( $CSVfp, 0, "," );
        $newCourt = $modx->newObject('myClassName');
        $newCourt->set('name', $data[0]);
        $newCourt->set('value', md5($data[0]));
        $newCourt->save();
	endwhile;
endif;
fclose( $CSVfp );

$clearfile = fopen($csvPath, "w");
$txt = "";
fwrite($clearfile, $txt);

fclose($clearfile);

My immediate problem is trying to figure out how to query the table to identify each record in the CSV file, match them against the existing records and skip them if duplicates are found.

I’ve been trying to search the table for other strings like this to understand how to test it, but only get one result back when i should’ve seen at least 3 containing the word Colorado:

$sql = 'SELECT *  FROM `modx_custom_table` WHERE `name` LIKE "%colorado%"';

$c = new xPDOCriteria($modx,$sql); // connect to the MODX DB
$html = '';
if ( $c->stmt && $c->stmt->execute() ) :
    $order_data = $c->stmt->fetch( PDO::FETCH_ASSOC );
    $names = explode('|', $order_data['name']); // string manipulations 
    
    foreach( $names as $name ) :
        $html .= "<li>$name</li>";
    endforeach;
    return $html;
endif;

Thanks for any advice

23

You have to call fetch in a loop or use fetchAll to read multiple rows:

$rows = $c->stmt->fetchAll(PDO::FETCH_ASSOC);
foreach ($rows as $r) {
	$html .= "<li>" . $r['name'] . "</li>";
}

Alternatively you could use xPDO for the whole query:

$rows = $modx->getCollection('myClassName', array('name:LIKE' => '%colorado%'));
foreach($rows as $r){
    $html .= "<li>" . $r->get('name') . "</li>";
}
24

I think using xPDO would be more appropriate.

25

i will try other things too.. .

26

Are you wanting to always trigger this manually, or automatically? If you set the snippet to be a static file, and name it with “.php” on the end, then you could add some code at the top to check for the existence of $modx, and if it doesn’t exist add the three or four lines of code that will initialize MODX inside the snippet. Then this file can be run by itself using CRON at automated intervals. Without $modx initialized inside the script, you have to run the snippet in a resource so that $modx is present.

27

If you want to automate the process, you could put the code in a plugin connected to some common event (which event depends on how often you want the code to run) – OnManagerLogin would happen less often than OnHandleRequest, for example.

The plugin could check the date and time and store the date somewhere (like a custom System Setting), and use that data to make sure it only runs once per week, per day, or per hour.

If the event in OnHandleRequest and it’s set to run once per day, for example, it would only execute for the first request after midnight each day.

As snowcreating suggested, you could also create a cron job to instantiate MODX and run the code at a certain interval.

You could also just have the file run by your cron job use cURL to visit the page where you put the snippet tag.

28

If you have this working in a snippet (or raw PHP) already and want to run it via cronjob, take a look at Scheduler:

https://docs.modmore.com/en/Open_Source/Scheduler/index.html

29

Actually, I just checked out the CronManager extra, and it works well without having to save the snippet as a static file or initialize MODX inside of it.