Large ROQL queries and leaking memory. Best practices? (1 Comment)

Topic by Adam

Content

Sooo.. I've been refactoring my code for days now, and I finally have to admit that I'm licked. I'm attempting to iterate through my Organizations table and "clean up" the SLAs. My logic is sound, it functions fine in a small environment. The issue only occurs in large queries.. the dreaded: Out of memory.

It starts out fine, but as you can tell from the logging -- the little piggy gobbles up all the dessert:

(19:21:46) Begin processing
(19:21:46) Processing rows 1 to 10000
(19:21:50) ** COMMIT ** :: Changes: 0, Rows: 500, Memory: 68943872
(19:21:53) ** COMMIT ** :: Changes: 0, Rows: 1000, Memory: 120061952
(19:21:55) ** COMMIT ** :: Changes: 0, Rows: 1500, Memory: 144703488
(19:21:58) ** COMMIT ** :: Changes: 0, Rows: 2000, Memory: 176422912
(19:22:00) ** COMMIT ** :: Changes: 0, Rows: 2500, Memory: 219938816
(19:22:03) ** COMMIT ** :: Changes: 0, Rows: 3000, Memory: 248774656
(19:22:05) ** COMMIT ** :: Changes: 0, Rows: 3500, Memory: 278921216

Fatal error: Out of memory (allocated 285736960) (tried to allocate 54 bytes) in /cgi-bin/site.cfg/scripts/custom/sla_fix.php on line 46

I've lost count of the number of different (sometimes completely ridiculous) ways I've attacked this. My fall-back is to add an offset argument at runtime and process in blocks rather than letting the script loop (this, however, is admitting defeat -- which leaves a bad taste in my mouth).

So, I submit to the wiser amongst us -- what am I doing wrong? Is my memory (mis)management really that bad? Is this to simply be chalked up "PHP SUX"? Is this an internal class memory leak, completely out of my control? ... Solar flares? :(

Version

August 2014 SP3

Code Block

<?

/**
 * sla_fix.php
 * 
 * Fix SLAs. Duh.
 * 
 * @author Adam
 * @version 1.0
 */

/** 
 * Initialization
 */

error_reporting(E_ALL & ~E_NOTICE);
ini_set("display_errors", 1);
ini_set('html_errors', false);

if (!defined('DOCROOT'))
{
    define('DOCROOT', get_cfg_var('doc_root'));
}

/**
 * Agent Authentication
 */

require_once (DOCROOT . '/include/services/AgentAuthenticator.phph');
// $account = AgentAuthenticator::authenticateSessionID();
use RightNow\Connect\v1_2 as RNCPHP;
/**
 * SLA Names
 */

define('SLA_NAME_SITE_ENGINEERING', 32);
define('SLA_NAME_DEALER_CSP',           33);
define('SLA_NAME_PRIVILEGED',           34);
define('SLA_NAME_INT_DEALER',           35);
define('SLA_NAME_SITE_SALES',       36);
define('SLA_NAME_SITE_SUPPORT',     37);
define('SLA_NAME_SITE_OP',          39);

/**
 * SLA Sets
 */

define('SLA_SET_DEALER_CSP',            1);
define('SLA_SET_SITE_SUPPORT',      2);
define('SLA_SET_SITE_SALES',        3);
define('SLA_SET_INT_DEALER',            4);
define('SLA_SET_PRIVILEGED',            5);
define('SLA_SET_SITE_ENGINEERING',  6);
define('SLA_SET_SITE_OP',           7);

/**
 * SLA States
 */

define('SLA_STATE_INACTIVE',    1);
define('SLA_STATE_ACTIVE',        2);
define('SLA_STATE_EXPIRED',        3);
define('SLA_STATE_DISABLED',    4);

/**
 * Configuration
 */

define('API_MAX_RESULTS', 10000);

/**
 * Globals
 */

$results   = 0;
$processed = 0;

$sla_map = array(
    SLA_SET_DEALER_CSP       => array(SLA_NAME_DEALER_CSP, 1, 31, 16, 7, 11),
    SLA_SET_SITE_SUPPORT     => array(SLA_NAME_SITE_SUPPORT, 28, 2, 8, 12, 14),
    SLA_SET_SITE_SALES       => array(SLA_NAME_SITE_SALES, 29, 3, 9, 15, 13),
    SLA_SET_INT_DEALER       => array(SLA_NAME_INT_DEALER, 17, 4, 10),
    SLA_SET_PRIVILEGED       => array(SLA_NAME_PRIVILEGED, 18, 5, 6), 
    SLA_SET_SITE_ENGINEERING => array(SLA_NAME_SITE_ENGINEERING, 19, 21, 24, 25),
    SLA_SET_SITE_OP          => array(SLA_NAME_SITE_OP, 20, 26, 30, 38));

/**
 * Utility Functions
 */

function recursive_array_search($needle, &$haystack)
{
    foreach($haystack as $key => $value)
    {
        $current_key = $key;
        if($needle === $value OR (is_array($value) && recursive_array_search($needle, $value) !== false))
        {
            return $current_key;
        }
    }

    return false;
}

function printMessage($string)
{
    print("(" . date("H:i:s") . ") " . $string . "\n");
}

/**
 * Code
 */

printMessage("Begin processing");

try
{
    $processed = 0;
    $paging = true;

    $current_id = $results;

    while($paging == true)
    {
        // Query all non-terminated Organizations
        $rrs = RNCPHP\ROQL::queryObject('SELECT Organization from Organization WHERE (CustomFields.c.terminated IS NULL OR CustomFields.c.terminated <> 1) AND ID > ' . $current_id . ' ORDER BY ID LIMIT ' . API_MAX_RESULTS);
        $rs = $rrs->next();

        $results += $rs->count();
        printMessage("Processing rows " . ($results-$rs->count()+1) . " to " . $results);

        while($row = $rs->next())
        {
            $changes = 0;
            $active_slas = array();

            $processed++;

            // Iterate through SLAs
            foreach($row->ServiceSettings->SLAInstances as $key => &$slai)
            {
                if(!in_array($slai->SLASet, $active_slas) && ($slai->NameOfSLA->ID !== $sla_map[recursive_array_search($slai->NameOfSLA->ID, $sla_map)][0])&& $slai->StateOfSLA == SLA_STATE_ACTIVE)
                {
                    print("Changing " . $slai->NameOfSLA->ID . " to " . $sla_map[recursive_array_search($slai->NameOfSLA->ID, $sla_map)][0] . "\n");

                    // Create new SLA instance
                    $sla1 = new RNCPHP\SLAInstance();
                    $sla1->ActiveDate = time();
                    $sla1->StateOfSLA = SLA_STATE_ACTIVE;
                    $sla1->NameOfSLA = $sla_map[recursive_array_search($slai->NameOfSLA->ID, $sla_map)][0];

                    // Add to array to prevent dupes
                    $active_slas[] = $sla_map[$slai->SLASet];

                    // Add new SLA instance
                    $row->ServiceSettings->SLAInstances[] = $sla1;
                    $sla1 = null;

                    $changes++;
                }

                // Unset all other SLAs
                $row->ServiceSettings->SLAInstances->offsetUnset((int)$key);
            }

            // Commit every 500 changes and/or 500 rows
            if(($changes === 500) || $processed % 500 === 0)
            {
                printMessage("** COMMIT ** :: Changes: " . $changes . ", Rows: " . $processed . ", Memory: " . memory_get_peak_usage(true));
                RNCPHP\ConnectAPI::commit();

                $changes = 0;
            }

            $slai = null;

            // Save changes (if any)
            if(!empty($active_slas))
            {
                printMessage("** SAVE ** :: Changes: " . $changes . ", Memory: " . memory_get_peak_usage(true));
                $row->save(RNCPHP\RNObject::SuppressAll);
            }

            $row = null;
            $active_slas = null;
        }

        printMessage("Finished processing rows " . (max(($results - API_MAX_RESULTS), 0)+1) . " to " . $results . ", Memory: " . memory_get_peak_usage(true));
        $paging = ($rs->count() % API_MAX_RESULTS) == 0;
    }
}
catch(Exception $e)
{
    printMessage("General error.\n Error Message: " . $e->getMessage());
}

$sla_map = null;
printMessage("End processing");


?>

Large ROQL queries and leaking memory. Best practices? (1 Comment)

Content

Version

Code Block

Trending Articles

Practice Sheet of Right form of verbs for HSC Students

Download: FK ft Shenky – Nakuyewa ”Prod by: Shenky”

How to win at Markstrat (Markstrat Tips and Tricks) – Vodites

Ominde Commission Report and Recommendations – Ominde Report of 1964

Bureau of Internal Revenue: Regional Offices (Directory)

GO 53 on Enhancement of Ex-gratia upto 5 Lakhs Toddy Tappers in Telangana

Cakewalk CA-2A Leveling Amplifier v2.0.1.97 WiN, v2.0.1.96 OSX Incl Keygen

Mp3 Download: Mdu - Kunjenjenjena

How the kill the job , when DTP request running for long hours.

Microsoft Intune から展開しているアプリのアップデートについて

18-year-old girl was beaten for half an hour by two Northampton men in 'an...

Car crash in Dunton Bassett leaves driver in critical condition

Macky 2, Two Others In Road Accident

Application log 00000000000000089514: Could not convert queue DLVST90CLNT

Detroit mafia: D’Anna Brothers agree to plea deal

Delivery block field greyed out using VA02

Muloraki Au

【個人撮影】スマホのプライベート映像♪「中に出さないで///」カラオケ屋での生ハメ撮りが流出ｗ【リベンジポルノ】＠PornHub

BREAKING NEWS: Diamond Platnumz Is Reported Dead After Ghastly Car Accident

FIAT 500 B0111 B0112