Issues with Xodus not downloading exports with over about 8,000 users

As the title says, Xodus will not create and download a .csv of more than about 8000 users, a file of about 19M (18891684b actually) the file contains all 11,865 users but for some reason above 8000 no go. Which I can change with this function:

public function beforeQuery() {
	$this->setProperty('limit', 0); // change to 8000
	return true;
}
  • PHP file/timeout/execution and memory settings are not the culprit (10 minutes and 1024M memory)
  • I can download and open the file Xodus creates manually
  • no (PHP) errors in webserver logs

it gets to the point in getfile.class.php:

public function outputArray(array $users, $count = false){

	$this->modx->log(modX::LOG_LEVEL_ERROR,'XODUS: RUNNING outputArray');

	if ($count === false) { $count = count($users);}

	if($count == 0){
		return '{"success":true,"message":"'.$this->modx->lexicon('xodus.export_no_users').'","object":{"total":"0"},"data":[]}';	
	}else{

		$action_url = $this->getActionURL();

		$this->modx->log(modX::LOG_LEVEL_ERROR,'XODUS: outputArray:: Got action url = ' . $action_url);

		if($this->fext == 'csv'){
			$this->modx->log(modX::LOG_LEVEL_ERROR,'XODUS: outputArray fext = CSV'.'{"success":true,"message":"'.count($users).' '.$this->modx->lexicon('xodus.export_users').'","object":{"file":"'.$this->timestamp.'.csv","action_url":"'.$action_url.'","total":"'.count($users).'"},"data":[]}');

			return '{
					"success":true,
					"message":"'.count($users).' '.$this->modx->lexicon('xodus.export_users').'",
					"object":{
						"file":"'.$this->timestamp.'.csv",
						"action_url":"'.$action_url.'",
						"total":"'.count($users).'"
						},
					"data":[]
				}';

		}else{
			...else....	
		}
	}
}

So I have added some debugging, and it will output the return value (correctly) to the modx log. but the actual php force download function never fires?

public function render(){

	$this->modx->log(modX::LOG_LEVEL_ERROR,'XODUS RUNNING CONTROLLER GETFILE');

	$this->failure('');
	$this->loadHeader = false;
	$this->loadFooter = false;
	$this->isFailure = false;
	$this->failureMessage = '';
	$this->content = '';
	$this->modx->lexicon->load('xodus:default');
	$core_path = dirname(dirname(__FILE__));
	$file_path = $core_path.'/tmp/'.$_GET['f'];
	if(file_exists($file_path)){
		$filepts = explode('.',$_GET['f']);
		$ext = $filepts[1];
		$mimetype = '';
		switch($ext){
			case 'csv':
				$mimetype = 'text/csv';
				break;
			case 'xls':
				$mimetype = 'application/excel';
				break;
			case 'xlsx':
				$mimetype = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet';
				break;	
		}

		$this->modx->log(modX::LOG_LEVEL_ERROR,'XODUS RUNNING GETFILE/SIZE = ' . filesize($file_path));
		header("Content-type: application/force-download");  
        header('Content-Disposition: inline; filename="'.$_GET['f'].'"');  
        header("Content-Transfer-Encoding: Binary"); 
	 	header("Content-length: ".filesize($file_path));  
        header('Content-Type: '.$mimetype);  
        header('Content-Disposition: attachment; filename="'.$_GET['f'].'"');  
        readfile($file_path);  
		unlink($file_path);
		exit();
	}else{
		 $this->modx->log(modX::LOG_LEVEL_ERROR,'XODUS: XodusGetfileManagerController::render'.print_r($scriptProperties,TRUE));
		$this->modx->lexicon->load('xodus:default');
		$this->content = '<html><title>'.$this->modx->lexicon('xodus').' '.$this->modx->lexicon('xodus.error').'</title><head></head><body>
		<h1>'.$this->modx->lexicon('xodus').' '.$this->modx->lexicon('xodus.error').'</h1>
		<p>'.$this->modx->lexicon('xodus.error_text').'. <a href="javascript:history.go(-1);">'.$this->modx->lexicon('xodus.return_link_text').'</a>.</p>
		</body></html>';
		
	return $this->content;
	}
}

You will get the first log entry in that function IF I set it to export less than 8000 users.

Anyone have any idea what might be happening here?? Is modx or ExtJS somehow blocking large files or something silly? Spent an entire day on this with no progress :frowning:

HELP!

For Fun, here is the modx log with my debug lines…

[2019-08-14 19:52:49] (ERROR @ core/components/xodus/processors/mgr/xodus/getfile.class.php : 208) XODUS: RUNNING DO EXPORT FILE

[2019-08-14 19:52:49] (ERROR @ core/components/xodus/processors/mgr/xodus/getfile.class.php : 222) XODUS: doCSV 11857

[2019-08-14 19:52:49] (ERROR @ core/components/xodus/processors/mgr/xodus/getfile.class.php : 230) XODUS FILENAME = core/components/xodus/tmp/user_export_1565812369

[2019-08-14 19:52:50] (ERROR @ core/components/xodus/processors/mgr/xodus/getfile.class.php : 347) XODUS: doCSV FPUT COMPLETE = 12640 / 11857

[2019-08-14 19:52:50] (ERROR @ core/components/xodus/processors/mgr/xodus/getfile.class.php : 164) XODUS: RUNNING outputArray

[2019-08-14 19:52:50] (ERROR @ core/components/xodus/processors/mgr/xodus/getfile.class.php : 471) XODUS: RUNNING getActionURL

[2019-08-14 19:52:50] (ERROR @ core/components/xodus/processors/mgr/xodus/getfile.class.php : 482) XODUS: action_url = ?a=getFile&namespace=xodus

[2019-08-14 19:52:50] (ERROR @ core/components/xodus/processors/mgr/xodus/getfile.class.php : 174) XODUS: outputArray:: Got action url = ?a=getFile&namespace=xodus

[2019-08-14 19:52:50] (ERROR @ core/components/xodus/processors/mgr/xodus/getfile.class.php : 177) XODUS: outputArray fext = CSV{“success”:true,“message”:“11857 users exported successfully. Your download is starting…”,“object”:{“file”:“user_export_1565812369.csv”,“action_url”:"?a=getFile&namespace=xodus",“total”:“11857”},“data”:
}

It could be the browser. I think Chrome limits downloaded files to about 30mb, though I would think it would crap out before the download starts.

This post suggests a problem Xodus and the php.ini memory_limit.

Would zipping the file help?

Technically it is the browser - ExtJS has a 30 second timeout, so it’s timing out on the client side… I actually exchanged a couple emails with Mark (H) & he confirmed that … I can tail the server/modx logs and watch the php script complete execution & send back the ‘success’ json response. Modx/ExtJS is no longer listening. (It takes 60 seconds or more for PHP to complete)

The file size is well under 30M when complete, around 18 or 19.

So the issue is now figuring out how to extend that ExtJS timeout within the Xodus scripts…