1 Replies - 1653 Views - Last Post: 08 May 2012 - 07:04 AM Rate Topic: -----

#1 Duckington  Icon User is offline

  • D.I.C Addict

Reputation: 172
  • View blog
  • Posts: 615
  • Joined: 12-October 09

ftp script error then timeouts everything

Posted 08 May 2012 - 06:42 AM

I've got a script which I've written which will be used as a cron script when it eventually works, but i'm having problems with it.

What it does:
- Connects via FTP to a server which contains a folder of images.
- Loops through those images (format of: userid.jpg)
- Checks if that userid exists in our database, if not deletes the image and continues
- If it does exist, transfers the image to a temp folder, then checks it against the user's stored image and see if they match
- If they match, delete from FTP and continue
- If they don't match, resize it down and update the user's image, then delete from FTP server and continue
- At end of script, run a clean up and delete all temp images

The problem:
- It works fine on my localhost, but when I upload it to our development server I run into issues:
- There is a limit of 500 images at a time (originally it was 1000, i've also tried 100 - all of these work on localhost)
- Sometimes (on edvelopment server) it works fine and it loops through all 500 images and it's done in about 6/7 seconds. But then sometimes it stops (at a random image it seems, e.g. last time it was something like 375, time before that was 100 & something), then it sits there no outputting anything for about 2 minutes, then continues looping through the rest giving me FTP and MySQL errors:

The first error I get (if for example it stops working on iteration #185) is:

LOOP :: [185]...
WARNING: [/var/www/html/local/ebs_images.php:297] ftp_get() [function.ftp-get]: Opening BINARY mode data connection for 140171.jpg(11296 bytes)....
ERR :: Could not transfer 140171.jpg...



Then for all the rest of the iterations, I get:

LOOP :: [num]...
MySQL server has gone away

SELECT * FROM mdl_user WHERE `username` = '140172'

WARNING: [/var/www/html/local/ebs_images.php:422] ftp_delete() [function.ftp-delete]: Can't open data connection....



So it seems that it runs into a problem on one file, stops for several minutes and then times everything out, ftp and mysql... But I have no idea why, because if i run that again, that file which it stopped on will be processed perfectly fine and not give me any errors...

Basically I'm at a loss as to what is causing it, i've tried adding in various time limits and settings to stop it timing out, but it seems that it's only timing out because it's running into a problem transfering a particular file (a problem which goes away next time it is run!?!?!).


The script is thus:

<?php
// Set error reporting to all
error_reporting(E_ALL);

// Set default timezone
date_default_timezone_set('Europe/London');

// Time limit
set_time_limit(180); # Max 3 mins, though shouldn't take that long to do 1000 hopefully. Closer to 30 secs.
ini_set('mysql.connect_timeout', 200);
ini_set('default_socket_timeout', 200);



// Require the Configuration File
require_once '../config.php';

// Set gd.jpeg_ignore_warning to true to avoid a warning when trying to create thumbnail of image
ini_set('gd.jpeg_ignore_warning', 1);

// Set up array of configuration variables
$CONFIG = array();
// String to contain all the output sent to the browser, so it can be emailed if we want
$CONFIG['output_text'] = '';

$start = explode(" ", microtime());
$start = $start[1] + $start[0];

///////////////////////////////////// EBSImages Class /////////////////////////////////////

class EBSImages
{
    
    const ftpServer = '***';
    const ftpUser = '***';
    const ftpPass = '***';
    const picsDir = 'moodle_pics';
    const limit = 500; # The max number of iterations it will do through the moodle_pics folder at a time
                        # There are 48,000 images, so can take a while.  Break it down into 1000 at a time
    
    private $connID;
    
    /**
     * Construct object of EBSImages and set up initial variable states
     */
    function EBSImages(){
        $this->connID = null;
    }
    
    /**
     * Run whole script
     */
    function run(){
        
        // Connect to EBS
        if( !$this->connectEBS() ){
            $this->end();
            echo $CONFIG['output_text'];
            exit('CANNOT CONNECT');
        }
        
        // Get all files
        if( !$this->beginTransfer($this->connID) ){
            $this->end();
            echo $CONFIG['output_text'];
            exit('CANNOT TRANSFER');
        }
        
        // Disconnect and run clean ups
        $this->end();
        
    }
    
    /**
     * Connect to the EBS server and point to moodle_pics directory
     * @return bool True or False value
     */
     private function connectEBS(){
               
        $connID = ftp_connect(self::ftpServer, 21, 180);
        
        if(!$connID){
            ctrace("FTP :: Connection has failed!");
            return false;
        }
        
        $this->connID = $connID;
        
        ctrace("FTP :: Connection established!");
        
        $loginResult = ftp_login($connID, self::ftpUser, self::ftpPass); 
        
        if(!$loginResult){
            ctrace("FTP :: Authentication has failed!");
            return false;
        }
        
        ctrace("FTP :: Authenticated!");
        ctrace("FTP :: Current directory: " . ftp_pwd($connID));
        
        if(!$this->switchDirectory($connID, self::picsDir)){
            return false;
        }
        
	return true;
        
    }
    
    private function switchDirectory($connID, $dir)
    {
        
        if( ftp_chdir($connID, $dir) ){
            ctrace("FTP :: Directory changed to: " . ftp_pwd($connID));
            return true;
        }
        else
        {
            ctrace("FTP :: Could not switch directory to /{$dir}");
            return false;
        }
        
    }
    
    /**
     * Begin the file transfer
     * Connect to EBS
     * Begin to loop through all the files in the directory
     */
    private function beginTransfer(){
        
       global $CFG;
        
       // Get all the contents in the working directory
        
       $contents = ftp_nlist($this->connID, ".");
       
       // REMOVE THIS LIMIT AND USE CONST WHEN FINISHED
       $limit = self::limit;
       $n = 0;
              
       foreach($contents as $file)
       {
           
           ctrace();
           
           if($n > $limit){
               return true;
           }
           
           ctrace("LOOP :: [$n]");
                      
           // Get moodle id of studentID/username
           $SID = $this->getSIDFromImageTitle($file);
           
           if(!$ID = $this->checkStudentIsValid($SID)){
               $n++;
               ctrace("ERR :: Invalid User {$SID} (Not in Moodle)");
               $this->deleteFromEBS($file);
               continue;
           }
           
           // We have an ID number now then or it would have returned false
           
           // Download & remove the file from EBS and store it in temp_pics directory
           $localPath = "temp_pics/".$file;
           
           if(!$this->downloadImage($file, $localPath)){
               $n++;
               # Won't delete for now, so we can try again later.
               continue;
           }
           
           // Scale the image down to 100x100
           // SimpleImage class doesn't incorporate a return true/false policy, so can only assume true
           $this->scaleImage($localPath, 100, 100);
           
           // Get a SHA1_FILE hash of the scaled image
           $hash = sha1_file($localPath);
           ctrace("IMG :: Image Hash: {$hash}");
           
           // Navigate to that user's directory within moodledata and see if there is an image there
           $dataPath = $CFG->dataroot . "/user/" . $this->workOutUserDirectory($ID) . "/" . $ID . "/";
           $dataF1 = $dataPath . "f1.jpg";
           $dataF2 = $dataPath . "f2.jpg";
           
           ctrace("IMG :: Navigating to " . $dataPath);
           
           // See if there is a file there called "f1.jpg" which will be the 100x100 version.
           if(!file_exists($dataF1)){
               
               ctrace("IMG :: f1.jpg does not exist, so transfering images across");
               
               // NEED TO MAKE SURE DIRECTORY EXISTS AND CREATE IT IF IT DOESN'T
               $strippedDownUserDataPath = substr_replace($dataPath, '', -1, 1); # Removes the "/" from the end
               $this->checkDataPathsExist($CFG->dataroot . "/user/" . $this->workOutUserDirectory($ID), $strippedDownUserDataPath);    
               
               // Transfer temp file across and save as f1.jpg
               rename($localPath, $dataF1);
               ctrace("IMG :: f1.jpg saved");
                   
               // Open newly saved f1.jpg and scale it down to 35x35 and save as f2.jpg
               $this->scaleImage($dataF1, 35, 35, $dataF2);
               ctrace("IMG :: f2.jpg saved");
                   
           }
           else
           {
               
               ctrace("IMG :: f1.jpg exists, so comparing");
               
               $dataHash = sha1_file($dataF1);
               
               ctrace("IMG :: Image Hash: {$dataHash}");
               
               // If hashs are equal, we don't need to do anything
               if($this->compareImages($hash, $dataHash)){
                   ctrace("IMG :: Images match.");
               }
               else
               {
                   
                   // Files are not the same, so we want to firstly transfer over the one from temp_pics
                   // and save it as f1.jpg to overwrite the old one. Then save a scaled down (35x35)
                   // version as f2.jpg to overwrite that.
                   ctrace("IMG :: Images do not match. Updating them now");
                   
                   // Move file
                   rename($localPath, $dataF1);
                   ctrace("IMG :: f1.jpg updated");
                   
                   // Open newly saved f1.jpg and scale it down to 35x35 and save as f2.jpg
                   $this->scaleImage($dataF1, 35, 35, $dataF2);
                   ctrace("IMG :: f2.jpg updated");
                   
               }
               
               
           }
           
           // Regardless of whether we updated or not, now delete the file from EBS
           // Otherwise when script continues to run during night, it will keep doing the same ones.
           $this->deleteFromEBS($file);

           $n++;
           
                     
       }
        
    }
    
    
    
    /**
     * Take a file name and remove the file extension, to get the SID number
     * @param type $imgTitle 
     * @return string SID number
     */
    private function getSIDFromImageTitle($imgTitle){
               
        // Strip extension from file
        $username = str_replace(".jpg", "", $imgTitle);
        
        return $username;
        
    }
    
    /**
     * Check if a student is valid within Moodle:
     *  - Exists
     *  - Logged in within n months
     *  - etc...
     * @param int $sid Student ID number
     * @return bool True or False value
     */
    private function checkStudentIsValid($SID){
        
        // Find ID number
        $check = get_record_select("user", "`username` = '{$SID}'");
        
        if(!isset($check->id)){
            return false;
        }
        
        return $check->id;
        
    }
    
    /**
     * Transfer an image from EBS to Moodle, via FTP
     * @param string $remotePath Path to image file on EBS
     * @param string $localPath Path to image file on Moodle (temp_pics)
     * @return bool True or False value based on result
     */
    private function downloadImage($remotePath, $localPath){
        
        // If successful, remove the old file from EBS
        if( ftp_get($this->connID, $localPath, $remotePath, FTP_BINARY) ){
            ctrace("IMG :: " . $remotePath . " successfully transfered to " . $localPath);
            return true;
        }
        else
        {
            ctrace("ERR :: Could not transfer " . $remotePath);
            return false;
        }
        
    }
    
    /**
     * Scale an image file.
     * The files from EBS are 200x200 but moodle contains 2 files: 100x100 and 35x35.
     * So we want to initially scale to 100x100 to perform a comparison, then if we need to update the image
     * we will scale another version down to 35x35 as well
     * @param string $imgFile
     * @param int $width
     * @param int $height 
     * @prerequisites Server must have Imagick package enabled
     */
    private function scaleImage($imgFile, $width, $height, $newName=null){
        
        $image = new SimpleImage();
        $image->load($imgFile);
        $image->resize($width, $height);
        
        if(!is_null($newName)){
            $image->save($newName);
        }else{
            $image->save($imgFile);
        }
        
        
        // Must assume it was successful
        ctrace("IMG :: Image successfully resized to {$width}x{$height}");
        
    }
    
    /**
     * Moodledata breaks users' directories down into sub-directories of overall parents
     * Eg. User "654" would be in: moodledata/user/0/654/
     * Eg. User "34407" would be in: moodledata/user/34000/34407/
     * @param int id mdl_user id for the user
     */
    private function workOutUserDirectory($id)
    {
        
        // if less than 1000, it's in the "0" folder
        if($id < 1000){
            return "0";
        }
        else
        {
            // Otherwise we need to switch the last 3 characters in the number for "0"s
            // E.g. 1234 is in the "1000" folder. 34407 is in the "34000" folder
            $str = (string)$id;
            $str = substr_replace($str, "000", -3, 3);
            return $str;            
        }
        
    }
    
    /**
     * Check if the directory to the user image in moodledata exists or not, in case we need to transfer
     * @param string $pathToParent
     * @param string $pathToUser 
     */
    private function checkDataPathsExist($pathToParent, $pathToUser){
        
        // User directory exists, so just return true - all is well
        if( is_dir($pathToUser) ){
            return true;
        }
        
        // The parent directory exists, e.g. "34000", but the child "34407" doesn#t exist within it.
        // Create it
        elseif( is_dir($pathToParent) ){
            ctrace("DIR :: User directory does not exist, attempting to create");
            if( mkdir($pathToUser, 0777) ){
                ctrace("DIR :: User directory created");
                return true;
            }
        }
        
        // Neither of them exist, so create the parent and the user directories
        else
        {
            ctrace("DIR :: Parent user directory and user directory do not exist, attempting to create");
            if( mkdir($pathToParent, '0777') && mkdir($pathToUser, '0777') ){
                ctrace("DIR :: Directories created");
                return true;
            }
        }
        
        return false;
                
    }
    
    /**
     * Compare the sha1_file() result of 2 files to see if they match.
     *  If the $moodImg doesn't exist, return false as will need to transfer it across from EBS
     *  If they do match, return true as we then don't need to transfer across
     *  If they both exist but don't match, return false, as we need to transfer across from EBS, as that should be latest version
     * @param string $ebsImg Path to image file on EBS
     * @param string $moodImg Path to image file on Moodle (moodledata)
     * @return bool True or False value
     */
    private function compareImages($ebsImg, $moodImg){
        
        if($ebsImg === $moodImg){
            return true;
        }
        
        return false;
        
    }
    
    /**
     * Delete a file from the EBS server
     * @param type $img 
     */
    private function deleteFromEBS($img){
                        
        if( ftp_delete($this->connID, $img) ){
            ctrace("FTP :: {$img} deleted from EBS server");
            return true;
        }
        
        ctrace("FTP :: Could not delete {$img} from EBS server");
        return false;
        
    }
    
    /**
     * Delete all temp pics
     */
    private function runCleanUp(){
        
        $files = glob("temp_pics/*.jpg");
        
        // Walk through all the files and unlink them
        
        array_walk($files, create_function('$f', 'unlink($f);'));
        
        ctrace("SYS :: Temporary files cleaned up");
        
    }
    
    /**
     * End the transfer of files.
     * Disconnect from EBS
     * Do any logging such as emailing output of cron script or whatever
     */
    private function end(){
                
        ftp_quit($this->connID);
        
        $this->runCleanUp();
        
    }
    
        
    
    
}





///////////////////////////////////// End of EBSImages Class /////////////////////////////////////


///////////////////////////////////// SimpleImage Class /////////////////////////////////////

/*
* File: SimpleImage.php
* Author: Simon Jarvis
* Copyright: 2006 Simon Jarvis
* Date: 08/11/06
* Link: http://www.white-hat-web-design.co.uk/articles/php-image-resizing.php
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details:
* http://www.gnu.org/licenses/gpl.html
*
*/
 
class SimpleImage {
 
   var $image;
   var $image_type;
 
   function load($filename) {
 
      $image_info = getimagesize($filename);
      $this->image_type = $image_info[2];
      if( $this->image_type == IMAGETYPE_JPEG ) {
         $this->image = @imagecreatefromjpeg($filename);
      } elseif( $this->image_type == IMAGETYPE_GIF ) {
 
         $this->image = imagecreatefromgif($filename);
      } elseif( $this->image_type == IMAGETYPE_PNG ) {
 
         $this->image = imagecreatefrompng($filename);
      }
   }
   function save($filename, $image_type=IMAGETYPE_JPEG, $compression=75, $permissions=null) {
 
      if( $image_type == IMAGETYPE_JPEG ) {
         imagejpeg($this->image,$filename,$compression);
      } elseif( $image_type == IMAGETYPE_GIF ) {
 
         imagegif($this->image,$filename);
      } elseif( $image_type == IMAGETYPE_PNG ) {
 
         imagepng($this->image,$filename);
      }
      if( $permissions != null) {
 
         chmod($filename,$permissions);
      }
   }
   function output($image_type=IMAGETYPE_JPEG) {
 
      if( $image_type == IMAGETYPE_JPEG ) {
         imagejpeg($this->image);
      } elseif( $image_type == IMAGETYPE_GIF ) {
 
         imagegif($this->image);
      } elseif( $image_type == IMAGETYPE_PNG ) {
 
         imagepng($this->image);
      }
   }
   function getWidth() {
 
      return imagesx($this->image);
   }
   function getHeight() {
 
      return imagesy($this->image);
   }
   function resizeToHeight($height) {
 
      $ratio = $height / $this->getHeight();
      $width = $this->getWidth() * $ratio;
      $this->resize($width,$height);
   }
 
   function resizeToWidth($width) {
      $ratio = $width / $this->getWidth();
      $height = $this->getheight() * $ratio;
      $this->resize($width,$height);
   }
 
   function scale($scale) {
      $width = $this->getWidth() * $scale/100;
      $height = $this->getheight() * $scale/100;
      $this->resize($width,$height);
   }
 
   function resize($width,$height) {
      $new_image = imagecreatetruecolor($width, $height);
      imagecopyresampled($new_image, $this->image, 0, 0, 0, 0, $width, $height, $this->getWidth(), $this->getHeight());
      $this->image = $new_image;
   }      
 
}

///////////////////////////////////// End of SimpleImage Class /////////////////////////////////////


///////////////////////////////////// Global Functions /////////////////////////////////////

function flushPage()
{
    ob_end_flush(); 
    flush();
    ob_start();
}

/**
 * Write to the screen
 * @param string $str The text to write
 * @param bool $bool Boolean value of whether or not to add "..." to the end of the text
 */
function ctrace($str = null)
{
    global $CONFIG;
    
    if(is_null($str)){
        echo "<br>";
        $CONFIG['output_text'] .= "\n";
    }
    else
    {
        echo $str . "...<br>";
        $CONFIG['output_text'] .= $str . "\n";
    }
    
    flushPage();
}

/**
 * Set a custom error handler
 * @param type $no Error number
 * @param type $msg Error message
 * @param type $file File
 * @param type $line Line
 */
function errorHandler($no, $msg, $file, $line)
{
    
    // For suppressed warnings/notices
    if(error_reporting() === 0){
        return;
    }
    
    switch($no)
    {
        case E_WARNING:
            ctrace("WARNING: [{$file}:{$line}] {$msg}");
        break;
        case E_NOTICE:
            ctrace("NOTICE: [{$file}:{$line}] {$msg}");
        break;
        default:
            ctrace("UNKNOWN {$no}: [{$file}:{$line}] {$msg}");
        break;
        
    }
    
    return true;
    
}

///////////////////////////////////// End of Global Functions /////////////////////////////////////



set_error_handler("errorHandler");

$EBS = new EBSImages();
$EBS->run();

$finish = explode(" ", microtime());
$finish = $finish[1] + $finish[0];
ctrace("This script took ".($finish - $start)." seconds to load");

// Could email output here
$emailTo = "***,***";

// Store log file
$path = $CFG->dataroot . "/datauploads/ebsimages/";
$fileName = "ebsImagesCronOutput_" . date('YmdHis') . ".txt";
$fullPath = $path . $fileName;
$attachPath = "/datauploads/ebsimages/" . $fileName;

$handle = fopen($fullPath, "a+");
fwrite($handle, $CONFIG['output_text']);
fclose($handle);

// Email admin with output
$admins = explode(",", $emailTo);
foreach($admins as $emailAdmin){
    $admin = get_record_select("user", "`email` = '{$emailAdmin}'");
    email_to_user($admin, $admin, "EBS Images Cron: Output", "See attached file for output of EBS images cron", "", $attachPath, $fileName);
}

?>




Has anyone run into a similar issue before and/or can offer any ideas as to what the problem is?

Thanks.

This post has been edited by Duckington: 08 May 2012 - 06:43 AM


Is This A Good Question/Topic? 0
  • +

Replies To: ftp script error then timeouts everything

#2 Duckington  Icon User is offline

  • D.I.C Addict

Reputation: 172
  • View blog
  • Posts: 615
  • Joined: 12-October 09

Re: ftp script error then timeouts everything

Posted 08 May 2012 - 07:04 AM

Never mind, I think I may have fixed it. I needed to turn ftp passive mode on.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1