View Issue Details

IDProjectCategoryView StatusLast Update
0000081LDMud 3.3Runtimepublic2009-02-28 16:14
ReporterlarsAssigned Tozesstra  
PrioritynormalSeverityfeatureReproducibilityN/A
Status resolvedResolutionfixed 
Product Version3.3 
Target Version3.3.719Fixed in Version3.3.719 
Summary0000081: Better feedback on low memory situations.
DescriptionWhen the user/master/system reserve is freed (or reallocated), the master lfun low_memory() should be called, given as argument which reserves are currently unavailable. If the function returns 0, the standard low-memory handling (gc + reallocation attempt) is done. If the function returns 1, the driver assumes that the mudlib took care of things and will just attempt to reallocate the memory.

In addition, a soft max-malloc limit is required - again, if it is exceeded, the low_memory() lfun is called.
TagsNo tags attached.
Attached Files
low_memory.diff (33,505 bytes)   
Index: doc/driver/invocation
===================================================================
--- doc/driver/invocation	(Revision 2505)
+++ doc/driver/invocation	(Arbeitskopie)
@@ -183,9 +183,16 @@
         Giving this option results in smaller, but also more fragmented
         swapfiles, and the swap performance may degrade.
 
-      --max-malloc <size>
-        Restrict total memory allocations to <size> bytes.
+      --hard-malloc-limit <size>
+        Restrict total memory allocation to <size> bytes.
         A <size> of 0 or 'unlimited' removes any restriction.\n"
+        
+      --soft-malloc-limit <size>
+        This value gives a soft limit of the allocated memory (kind of low
+        watermark). If this value is exceeded, the driver will call low_memory()
+        in the master to inform the mudlib about the (potenntially) developing low
+        memory situation.
+        A <size> of 0 or 'unlimited' removes this threshold.
 
       --min-malloc <size>
       --min-small-malloc <size>
Index: doc/master/low_memory
===================================================================
--- doc/master/low_memory	(Revision 0)
+++ doc/master/low_memory	(Revision 0)
@@ -0,0 +1,58 @@
+SYNOPSIS
+        void low_memory(int what, int limitvalue, int memory, int reservestate)
+
+DESCRIPTION
+        This efun is called when there is a (potential) low-memory situation.
+        Two different limits for the amount of allocated memory can be
+        configured: a soft limit and a hard limit.
+        If the soft limit is exceeded, the driver calls low_memory() but does
+        nothing else.
+        If the hard limit is exceeded, the driver calls low_memory() as well,
+        but initiates a garbage collection directly after the call.
+        The efun is called as well directly before a user-initiated garbage
+        collection is started.
+
+        <what> denotes the type of limit which was exceeded:
+        - NO_MALLOC_LIMIT_EXCEEDED   (0)
+            No limit was exceeded, by a garbage collection was requested by a
+            call to garbage_collection() and it will begin right after
+            low_memory() returns.
+        - SOFT_MALLOC_LIMIT_EXCEEDED (1)
+            The soft limit was exceeded.
+        - HARD_MALLOC_LIMIT_EXCEEDED (2)
+            The hard limit was exceeded. A garbage collection will begin right
+            after low_memory() returns.
+
+        <limit> specifies the numerical value of the limit which was exceeded.
+        It is 0 in case of NO_MALLOC_LIMIT_EXCEEDED.
+
+        <memory> specifies the amount of allocated memory right now.
+
+        <reservestate> specifies the current state of the memory reserves of
+        the driver. The states of the 3 different reserves are OR'ed together.
+        - USER_RESERVE_AVAILABLE   (1)
+          The user reserve is available.
+        - MASTER_RESERVE_AVAILABLE (2)
+          The master reserve is available.
+        - SYSTEM_RESERVE_AVAILABLE (4)
+          The system reserve is available. 
+
+        This efun might be used to inform users about the garbage collection
+        and the expeced long lag.
+        Another possibility is to initiate a garbage collection deliberately
+        at a suitable time (e.g. during the following night) after the soft
+        limit was hit.
+
+REMARKS
+        If in a low_memory condition when the memory reserves have been used
+        already they could not be re-allocated after a garbage_collection()
+        slow_shut_down() is called instead of calling this function again.
+
+HISTORY
+        Introduced in 3.3.719
+
+SEE ALSO
+        debug_info(E), garbage_collection(E)
+        slow_shut_down(M)
+        memory(C)
+        malloc(D)
Index: doc/master/slow_shut_down
===================================================================
--- doc/master/slow_shut_down	(Revision 2505)
+++ doc/master/slow_shut_down	(Arbeitskopie)
@@ -38,3 +38,4 @@
 
 SEE ALSO
         quota_demon(M), notify_shutdown(M), shutdown(E), malloc(D), memory(C)
+        low_memory(M)
Index: doc/concepts/memory
===================================================================
--- doc/concepts/memory	(Revision 2505)
+++ doc/concepts/memory	(Arbeitskopie)
@@ -53,4 +53,6 @@
         other objects.
 
 SEE ALSO
-        clean_up(A), slow_shut_down(M), quota_demon(M), malloc(D)
+        clean_up(A), slow_shut_down(M), quota_demon(M), low_memory(M)
+        malloc(D),
+        garbage_collection(E)
Index: src/settings/lethynna
===================================================================
--- src/settings/lethynna	(Revision 2505)
+++ src/settings/lethynna	(Arbeitskopie)
@@ -40,7 +40,7 @@
 with_htable_size=65536
 with_otable_size=8192
 
-with_max_malloced=0x20000000
+with_hard_malloc_limit=0x20000000
 
 with_max_array_size=10000
 with_max_mapping_keys=40000
Index: src/settings/evermore
===================================================================
--- src/settings/evermore	(Revision 2505)
+++ src/settings/evermore	(Arbeitskopie)
@@ -96,7 +96,7 @@
 with_reserved_user_size=800000
 with_catch_reserved_cost=0xF000
 with_master_reserved_cost=0xF000
-with_max_malloced=0
+with_hard_malloc_limit=0
 
 enable_use_system_crypt=no
 enable_with_pthreads=no
Index: src/settings/dt2
===================================================================
--- src/settings/dt2	(Revision 2505)
+++ src/settings/dt2	(Arbeitskopie)
@@ -32,7 +32,7 @@
 with_max_players=100
 
 with_min_malloced=0x1000000
-with_max_malloced=0x8000000
+with_hard_malloc_limit=0x8000000
 
 with_htable_size=65536
 with_itable_size=4096
Index: src/settings/gestrandet
===================================================================
--- src/settings/gestrandet	(Revision 2505)
+++ src/settings/gestrandet	(Arbeitskopie)
@@ -23,7 +23,7 @@
 with_reserved_user_size=0x400000
 with_reserved_master_size=0x100000
 with_reserved_system_size=0x200000
-with_max_malloced=0x1c000000
+with_hard_malloc_limit=0x1c000000
 enable_min_malloced=yes
 with_min_malloced=0x8000000
 enable_min_small_malloced=yes
Index: src/settings/eotl
===================================================================
--- src/settings/eotl	(Revision 2505)
+++ src/settings/eotl	(Arbeitskopie)
@@ -15,7 +15,7 @@
 with_read_file_max_size=3000000
 with_max_byte_transfer=500000
 
-with_max_malloced=0x6000000
+with_hard_malloc_limit=0x6000000
 with_min_malloced=0x2000000
 
 with_time_to_clean_up=7200
Index: src/settings/bat
===================================================================
--- src/settings/bat	(Revision 2505)
+++ src/settings/bat	(Arbeitskopie)
@@ -65,7 +65,7 @@
 with_compiler_stack_size=600
 with_max_user_trace=120
 with_max_trace=125
-with_max_malloced=0
+with_hard_malloc_limit=0
 
 with_max_cost=20000000
 with_catch_reserved_cost=150000
Index: src/settings/silberland
===================================================================
--- src/settings/silberland	(Revision 2505)
+++ src/settings/silberland	(Arbeitskopie)
@@ -38,4 +38,4 @@
 enable_use_parse_command=no
 enable_share_variables=no
 
-with_max_malloced=0x5000000
+with_hard_malloc_limit=0x5000000
Index: src/settings/beutelland
===================================================================
--- src/settings/beutelland	(Revision 2505)
+++ src/settings/beutelland	(Arbeitskopie)
@@ -18,7 +18,7 @@
 with_reserved_user_size=0x800000
 with_reserved_master_size=0x50000
 with_reserved_system_size=0x100000
-with_max_malloced=0x16000000
+with_hard_malloc_limit=0x16000000
 
 # --- Interpreter ---
 with_max_cost=1000000
Index: src/settings/dragonfire
===================================================================
--- src/settings/dragonfire	(Revision 2505)
+++ src/settings/dragonfire	(Arbeitskopie)
@@ -41,7 +41,7 @@
 --with-portno=1999 \
 --with-udp-port=2000 \
 --with-max-cost=2100000 \
---with-max-malloced=0 \
+--with-hard-malloc-limit=0 \
 --with-min-malloced=0x80000 \
 --with-min-small-malloced=0x80000 \
 --with-otable-size=8192 \
Index: src/settings/avalon
===================================================================
--- src/settings/avalon	(Revision 2505)
+++ src/settings/avalon	(Arbeitskopie)
@@ -40,7 +40,7 @@
 with_htable_size=65536
 with_otable_size=8192
 
-with_max_malloced=0x23000000
+with_hard_malloc_limit=0x23000000
 
 with_max_array_size=10000
 with_max_mapping_keys=40000
Index: src/settings/pkmud
===================================================================
--- src/settings/pkmud	(Revision 2505)
+++ src/settings/pkmud	(Arbeitskopie)
@@ -46,5 +46,5 @@
 enable_share_variables=no
 
 with_min_malloced=0x80000
-with_max_malloced=0x5000000
+with_hard_malloc_limit=0x5000000
 enable_dynamic_costs=yes
Index: src/settings/default
===================================================================
--- src/settings/default	(Revision 2505)
+++ src/settings/default	(Arbeitskopie)
@@ -127,7 +127,7 @@
 # This value gives the upper limits for the total allocated memory.
 # A value of 0 means 'unlimited'.
 
-with_max_malloced=0x4000000 
+with_hard_malloc_limit=0x4000000 
 
 # If these values are >0, the driver will reserve this amount of memory
 # on startup for large resp. small blocks, reducing the block fragmentation.
Index: src/settings/traumland
===================================================================
--- src/settings/traumland	(Revision 2505)
+++ src/settings/traumland	(Arbeitskopie)
@@ -31,7 +31,7 @@
 with_reserved_user_size=0x1000000
 with_reserved_master_size=0x100000
 with_reserved_system_size=0x400000
-with_max_malloced=0x10000000 
+with_hard_malloc_limit=0x10000000 
 with_min_malloced=0x1000000
 with_min_small_malloced=0x400000
 
Index: src/settings/unitopia
===================================================================
--- src/settings/unitopia	(Revision 2505)
+++ src/settings/unitopia	(Arbeitskopie)
@@ -30,7 +30,7 @@
 with_reserved_user_size=0x400000
 with_reserved_master_size=0x100000
 with_reserved_system_size=0x200000
-with_max_malloced=0x60000000
+with_hard_malloc_limit=0x60000000
 enable_min_malloced=yes
 with_min_malloced=0x8000000
 enable_min_small_malloced=yes
Index: src/settings/morgengrauen
===================================================================
--- src/settings/morgengrauen	(Revision 2505)
+++ src/settings/morgengrauen	(Arbeitskopie)
@@ -59,7 +59,7 @@
 with_max_byte_transfer=200000
 with_min_malloced=0xfa00000
 with_min_small_malloced=0x8200000
-with_max_malloced=0x7fffffff
+with_hard_malloc_limit=0x7fffffff
 with_pcre_recursion_limit=20000
 with_optimize=med
 
Index: src/settings/tubmud
===================================================================
--- src/settings/tubmud	(Revision 2505)
+++ src/settings/tubmud	(Arbeitskopie)
@@ -31,7 +31,7 @@
 
 # --- Memory ---
 with_reserved_user_size=0x100000
-with_max_malloced=0x06000000
+with_hard_malloc_limit=0x06000000
 
 # --- Interpreter ---
 enable_strict_euids=no
Index: src/settings/finalfrontier
===================================================================
--- src/settings/finalfrontier	(Revision 2505)
+++ src/settings/finalfrontier	(Arbeitskopie)
@@ -32,7 +32,7 @@
 with_reserved_user_size=0x400000
 with_reserved_master_size=0x100000
 with_reserved_system_size=0x200000
-with_max_malloced=0x16000000
+with_hard_malloc_limit=0x16000000
 enable_min_malloced=yes
 with_min_malloced=0x8000000
 enable_min_small_malloced=yes
Index: src/settings/wunderland
===================================================================
--- src/settings/wunderland	(Revision 2505)
+++ src/settings/wunderland	(Arbeitskopie)
@@ -106,7 +106,7 @@
 --with-malloc=default \
 --with-min-malloced=0 \
 --with-min-small-malloced=0 \
---with-max-malloced=0x6000000 \
+--with-hard-malloc-limit=0x6000000 \
 --with-total-trace-length=0x1000 \
 --with-wizlist-file=WIZLIST \
 $*
Index: src/smalloc.c
===================================================================
--- src/smalloc.c	(Revision 2505)
+++ src/smalloc.c	(Arbeitskopie)
@@ -3039,7 +3039,7 @@
                 )
                )
             {
-                static const char mess[] = "MAX_MALLOCED limit reached.\n";
+                static const char mess[] = "HARD_MALLOC_LIMIT reached.\n";
                 writes(2, mess);
 
                 ptr = NULL;
Index: src/xalloc.c
===================================================================
--- src/xalloc.c	(Revision 2505)
+++ src/xalloc.c	(Arbeitskopie)
@@ -126,10 +126,19 @@
 mp_int reserved_master_size = RESERVED_MASTER_SIZE;
 mp_int reserved_system_size = RESERVED_SYSTEM_SIZE;
 
-mp_int min_malloced       = MIN_MALLOCED;           /* Allocation limits */
-mp_int min_small_malloced = MIN_SMALL_MALLOCED;     /* Allocation limits */
-mp_int max_malloced       = MAX_MALLOCED;
+/* at startup reserve these amounts of memory for large and small blocks */
+mp_int min_malloced       = MIN_MALLOCED;
+mp_int min_small_malloced = MIN_SMALL_MALLOCED;
+/* this is the hard limit for memory allocations. */
+static mp_int max_malloced       = HARD_MALLOC_LIMIT_DEFAULT;
+/* this is a soft limit for memory allocations. It serves as a kind of low
+ * watermark. If exceeded, the game driver will inform the mudlib by calling
+ * low_memory() in the master. */
+static mp_int soft_malloc_limit  = SOFT_MALLOC_LIMIT_DEFAULT;
 
+/* Was the low_memory() already called in the master`*/
+static Bool low_memory_applied = FALSE;
+
 int stack_direction = 0; /*  0: Unknown stack behaviour
                           * +1: Stack grows upward
                           * -1: Stack grows downward
@@ -180,7 +189,7 @@
 /* Forward declarations */
 
 #ifdef MALLOC_LPC_TRACE
-static void write_lpc_trace (int d, word_t *p, int oneline);
+static void write_lpc_trace (int d, word_t *p, int oneline) __attribute__((nonnull(2)));
 #endif
 
 #ifdef GC_SUPPORT
@@ -518,7 +527,7 @@
 #ifndef NO_MEM_BLOCK_SIZE
     if (max_malloced > 0 && (mp_int)xalloc_stat.size > max_malloced)
     {
-        static const char mess[] = "MAX_MALLOCED limit reached.\n";
+        static const char mess[] = "HARD_MALLOC_LIMIT reached.\n";
         writes(2, mess);
         if (malloc_privilege < MALLOC_SYSTEM)
             return MY_TRUE;
@@ -1663,4 +1672,126 @@
     return p;
 } /* string_copy_traced() */
 
+/*-------------------------------------------------------------------------*/
+void
+notify_lowmemory_condition(short what)
+/* Calls low_memory(what, <limit>, <memory_consumption>, <reserves> ) 
+ * in the master.
+ */
+{
+    short reservestate = 0;
+    push_number(inter_sp, what);
+    if (what == SOFT_MALLOC_LIMIT_EXCEEDED)
+        push_number(inter_sp, soft_malloc_limit);
+    else if (what == HARD_MALLOC_LIMIT_EXCEEDED)
+        push_number(inter_sp, max_malloced);
+    else
+        push_number(inter_sp, 0);
+    push_number(inter_sp, xalloc_stat.size);
+    if (reserved_user_area)
+        reservestate |= USER_RESERVE_AVAILABLE;
+    if (reserved_master_area)
+        reservestate |= MASTER_RESERVE_AVAILABLE;
+    if (reserved_system_area)
+        reservestate |= SYSTEM_RESERVE_AVAILABLE;
+    push_number(inter_sp, reservestate);
+    callback_master(STR_LOW_MEMORY, 4);
+} /* inform_lowmemory_condition */
+
+/*-------------------------------------------------------------------------*/
+void
+check_for_soft_malloc_limit (void)
+/* If soft_malloc_limit is set, check if the allocated memory exceeds it.
+ * If yes and the master was not notified until now, low_memory() is called 
+ * in the mudlib master and low_memory_applied ist set to prevent any further
+ * notifications.
+ * If the limit is not exceeded, low_memory_applied is reset.
+ * Should be called from the backend.
+ */
+{
+#ifndef NO_MEM_BLOCK_SIZE
+    if (soft_malloc_limit > 0)
+    {
+        if ((mp_int)xalloc_stat.size > soft_malloc_limit)
+        {
+            if (!low_memory_applied)
+            {
+                /* call low_memory(malloced_memory) in the master but first
+                 * set the flag to prevent calling every backend cycle in case
+                 * of errors. */
+                low_memory_applied = TRUE;
+                notify_lowmemory_condition(SOFT_MALLOC_LIMIT_EXCEEDED);
+            }
+        }
+        else if (low_memory_applied)
+        {
+            /* OK, memory consumption shrunk below the soft limit. Reset the 
+             * warning, so that the next time the limit is exceeded,
+             * the master apply is done again. */
+            low_memory_applied = FALSE;
+        }
+    }
+#endif
+} /* check_for_soft_malloc_limit() */
+
+/*-------------------------------------------------------------------------*/
+Bool
+set_memory_limit(short what, mp_int limit)
+/* Sets the <what> memory limit to <limit>.
+ * <limit> for SOFT_MLIMIT has to be < HARD_MLIMIT.
+ * return TRUE on success and FALSE otherwise.
+ * If the current memory allocation is smaller than a new soft limit, the flag
+ * low_memory_applied is reset.
+ */
+{
+#ifndef NO_MEM_BLOCK_SIZE
+    // limits smaller than the sum of reserves (and some more) are harmful and
+    // lead anyway to immediate crashes... But we ignore this here, because 
+    // reserve_memory() will deal with the needed sizes for the reserves and the
+    // minimum allocations. And later on in the game we will just prevent to
+    // set the limit smaller then the already allocated memory.
+    
+    if (what == MALLOC_SOFT_LIMIT)
+    {
+        if (limit >= max_malloced || limit < 0)
+            return FALSE;
+        
+        soft_malloc_limit = limit;
+        /* reset flag if appropriate. */
+        if ((mp_int)xalloc_stat.size < soft_malloc_limit
+            && low_memory_applied)
+            low_memory_applied = FALSE;
+    }
+    else 
+    {
+        // setting the limit below the currently allocated memory seems to be 
+        // a very bad idea.
+        if (limit && limit <= (mp_int)xalloc_stat.size)
+            return FALSE;
+        
+        max_malloced = limit;
+        /* TODO: or disable it altogether? */
+        if (soft_malloc_limit > max_malloced)
+            soft_malloc_limit = max_malloced - 1 ;
+    }
+    return TRUE;
+#else
+    return FALSE;
+#endif // NO_MEM_BLOCK_SIZE
+} /* set_memory_limit */
+
+/*-------------------------------------------------------------------------*/
+mp_int
+get_memory_limit(short what)
+/* Return the value of limit <what>. */
+{
+#ifndef NO_MEM_BLOCK_SIZE
+    if (what == MALLOC_SOFT_LIMIT)
+        return soft_malloc_limit;
+    else
+        return max_malloced;
+#else
+    return 0;
+#endif
+}
 /***************************************************************************/
Index: src/main.c
===================================================================
--- src/main.c	(Revision 2505)
+++ src/main.c	(Arbeitskopie)
@@ -1040,7 +1040,8 @@
  , cHBInterval      /* --heart-interval     */
  , cHostname        /* --hostname           */
  , cHostaddr        /* --hostaddr           */
- , cMaxMalloc       /* --max-malloc         */
+ , cMaxMalloc       /* --hard-malloc-limit  */
+ , cSoftMallocLimit /* --soft-malloc-limit  */
  , cMaxArray        /* --max-array          */
  , cMaxBytes        /* --max-bytes          */
  , cMaxCallouts     /* --max-callouts       */
@@ -1364,13 +1365,22 @@
         "    Reuse free space in the swap file immediately.\n"
       }
 
-    , { 0,   "max-malloc",         cMaxMalloc,      MY_TRUE
-      , "  --max-malloc <size>\n"
-      , "  --max-malloc <size>\n"
-        "    Restrict total memory allocations to <size> bytes. A <size> of 0\n"
+    , { 0,   "hard-malloc-limit",  cMaxMalloc,      MY_TRUE
+      , "  --hard-malloc-lmit <size>\n"
+      , "  --hard-malloc-limit <size>\n"
+        "    Restrict total memory allocation to <size> bytes. A <size> of 0\n"
         "    or 'unlimited' removes any restriction.\n"
       }
 
+    , { 0,   "soft-malloc-limit",  cSoftMallocLimit,  MY_TRUE
+      , "  --soft-malloc-lmit <size>\n"
+      , "  --soft-malloc-limit <size>\n"
+        "    If total memory allocation exceeds <size> bytes, inform the mudlib\n"
+        "    master about a developing low memory situation. A <size> of 0\n"
+        "    or 'unlimited' removes the threshold. <size> must be smaller than\n"
+        "    --hard-malloc-limit.\n"
+      }
+
     , { 0,   "min-malloc",         cMinMalloc,      MY_TRUE
       , "  --min-malloc <size>\n"
       , "  --min-malloc <size>\n"
@@ -1972,12 +1982,18 @@
         , MIN_SMALL_MALLOCED
         );
 
-  printf("                 max allocation:       ");
-  if (MAX_MALLOCED > 0)
-      printf("%9d\n", MAX_MALLOCED);
+  printf("                 hard memory allocation limit: ");
+  if (HARD_MALLOC_LIMIT_DEFAULT > 0)
+      printf("%9d\n", HARD_MALLOC_LIMIT_DEFAULT);
   else
       printf("unlimited\n");
-
+  
+  printf("                 soft memory allocation limit: ");
+    if (SOFT_MALLOC_LIMIT_DEFAULT > 0)
+        printf("%9d\n", SOFT_MALLOC_LIMIT_DEFAULT);
+    else
+        printf("unlimited\n");
+    
   printf("Internal tables: shared string hash:      %6d entries\n"
          "                 object hash:             %6d entries\n"
          "                 reserved name hash:      %6d entries\n"
@@ -2513,19 +2529,33 @@
     case cMaxMalloc:
         if (!strcasecmp(pValue, "unlimited"))
         {
-            max_malloced = 0;
+            set_memory_limit(MALLOC_HARD_LIMIT, 0);
         }
         else
         {
-            max_malloced = strtol(pValue, (char **)0, 0);
-            if (max_malloced < 0)
+            if (!set_memory_limit(MALLOC_HARD_LIMIT, strtol(pValue, (char **)0, 0)))
             {
-                fprintf(stderr, "Illegal value '%s' for --max-malloc\n", pValue);
+                fprintf(stderr, "Illegal value '%s' for --hard-malloc-limit\n", pValue);
                 return hrError;
             }
         }
         break;
 
+    case cSoftMallocLimit:
+            if (!strcasecmp(pValue, "unlimited"))
+            {
+                set_memory_limit(MALLOC_SOFT_LIMIT, 0);
+            }
+            else
+            {
+                if (!set_memory_limit(MALLOC_SOFT_LIMIT, strtol(pValue, (char **)0, 0)))
+                {
+                    fprintf(stderr, "Illegal value '%s' for --soft-malloc-limit\n", pValue);
+                    return hrError;
+                }
+            }
+            break;
+            
     case cMudlib:
         if (chdir(pValue) == -1) {
             fprintf(stderr, "Bad mudlib directory: %s\n", pValue);
Index: src/xalloc.h
===================================================================
--- src/xalloc.h	(Revision 2505)
+++ src/xalloc.h	(Arbeitskopie)
@@ -67,10 +67,29 @@
 /* --- Constants --- */
 
 /* Allocation privilege levels */
+enum malloc_privileges {
+    MALLOC_USER    = 0,
+    MALLOC_MASTER  = 1,
+    MALLOC_SYSTEM  = 2,
+};
 
-#define MALLOC_USER    (0)
-#define MALLOC_MASTER  (1)
-#define MALLOC_SYSTEM  (2)
+/* memory limits */
+enum memory_limit_types {
+    MALLOC_SOFT_LIMIT  = 1,
+    MALLOC_HARD_LIMIT  = 2,
+};
+/* low memory conditions */
+enum low_memory_conditions {
+    NO_MALLOC_LIMIT_EXCEEDED    = 0,
+    SOFT_MALLOC_LIMIT_EXCEEDED  = MALLOC_SOFT_LIMIT,
+    HARD_MALLOC_LIMIT_EXCEEDED  = MALLOC_HARD_LIMIT,
+};
+/* Flags for available reserves */
+enum available_reserve_flags {
+    USER_RESERVE_AVAILABLE   = 1,
+    MASTER_RESERVE_AVAILABLE = 2,
+    SYSTEM_RESERVE_AVAILABLE = 4,
+};
 
 /* --- Variables --- */
 
@@ -84,7 +103,6 @@
 extern mp_int reserved_system_size;
 extern mp_int min_malloced;
 extern mp_int min_small_malloced;
-extern mp_int max_malloced;
 extern int stack_direction;
 
 
@@ -122,43 +140,49 @@
 #define xalloc_pass(size)       xalloc_traced((size) MTRACE_PASS)
 #define rexalloc_pass(old,size) rexalloc_traced((old),(size) MTRACE_PASS)
 
-extern size_t  xalloced_size (POINTER p);
+extern size_t  xalloced_size (POINTER p)  __attribute__((nonnull(1)));
 extern size_t  xalloc_overhead (void);
-extern POINTER xalloc_traced(size_t size MTRACE_DECL) 
-                                  MALLOC __attribute__ ((warn_unused_result));
+
+extern POINTER xalloc_traced(size_t size MTRACE_DECL)
+       MALLOC __attribute__ ((warn_unused_result));
+
 extern void    xfree(POINTER);
-extern POINTER rexalloc_traced(POINTER, size_t MTRACE_DECL)
-                                  MALLOC __attribute__ ((warn_unused_result));
-extern POINTER pxalloc_traced(size_t MTRACE_DECL) 
-                                  MALLOC __attribute__ ((warn_unused_result));
-extern POINTER prexalloc_traced(POINTER, size_t MTRACE_DECL)
-                                  MALLOC __attribute__ ((warn_unused_result));
+
+extern POINTER rexalloc_traced(POINTER, size_t size MTRACE_DECL) 
+       MALLOC __attribute__((warn_unused_result));
+
+extern POINTER pxalloc_traced(size_t size MTRACE_DECL)
+       MALLOC __attribute__ ((warn_unused_result));
+extern POINTER prexalloc_traced(POINTER, size_t size MTRACE_DECL)
+       MALLOC __attribute__((warn_unused_result));
+
 extern void    pfree(POINTER);
-extern void  * malloc_increment_size (void *vp, size_t size);
+extern void  * malloc_increment_size (void *vp, size_t size)
+                                __attribute__((nonnull(1)));
 
 #ifdef GC_SUPPORT
-extern void x_clear_ref (POINTER p);
-extern int x_mark_ref (POINTER p);
-extern Bool x_test_ref (POINTER p);
+extern void x_clear_ref (POINTER p)  __attribute__((nonnull(1)));
+extern int x_mark_ref (POINTER p)  __attribute__((nonnull(1)));
+extern Bool x_test_ref (POINTER p)  __attribute__((nonnull(1)));
 #endif /* GC_SUPPORT */
 
 #ifdef MALLOC_TRACE
 extern void store_print_block_dispatch_info(void *block, void (*func)(int, void *, int) );
-extern Bool is_freed(void *p, p_uint minsize);
+extern Bool is_freed(void *p, p_uint minsize) __attribute__((nonnull(1)));
 #endif /* MALLOC_TRACE */
 
 #ifdef CHECK_OBJECT_GC_REF
-extern void note_object_allocation_info ( void *block );
-extern void note_program_allocation_info ( void *block );
-extern Bool is_object_allocation ( void *block );
-extern Bool is_program_allocation ( void *block );
+extern void note_object_allocation_info ( void *block ) __attribute__((nonnull(1)));
+extern void note_program_allocation_info ( void *block ) __attribute__((nonnull(1)));
+extern Bool is_object_allocation ( void *block ) __attribute__((nonnull(1)));
+extern Bool is_program_allocation ( void *block ) __attribute__((nonnull(1)));
 #endif /* CHECK_OBJECT_RC_REF */
 
 /* Functions directly exported from the allocator: */
 
-extern void mem_dump_data(strbuf_t *sbuf);
-extern void mem_dump_extdata(strbuf_t *sbuf);
-extern void mem_dinfo_data(svalue_t *svp, int value);
+extern void mem_dump_data(strbuf_t *sbuf) __attribute__((nonnull(1)));
+extern void mem_dump_extdata(strbuf_t *sbuf) __attribute__((nonnull(1)));
+extern void mem_dinfo_data(svalue_t *svp, int value) __attribute__((nonnull(1)));
 extern void mem_consolidate (Bool force);
 extern Bool mem_dump_memory(int fd);
 #ifdef MALLOC_EXT_STATISTICS
@@ -178,13 +202,19 @@
 #define string_copy(s) string_copy_traced(s)
 #endif
 
-extern char * string_copy_traced(const char *str MTRACE_DECL) MALLOC;
-extern void dump_lpc_trace (int d, void *p);
-extern void dump_malloc_trace (int d, void *adr);
+extern char * string_copy_traced(const char *str MTRACE_DECL) MALLOC
+       __attribute__((nonnull(1))) __attribute__((warn_unused_result));
+extern void dump_lpc_trace (int d, void *p) __attribute__((nonnull(2)));
+extern void dump_malloc_trace (int d, void *adr) __attribute__((nonnull(2)));
 
 extern void get_stack_direction (void);
 extern void assert_stack_gap(void);
 extern void reserve_memory (void);
 extern void reallocate_reserved_areas(void);
+extern void check_for_soft_malloc_limit(void);
+extern void notify_lowmemory_condition(short what);
 
+extern Bool   set_memory_limit(short what, mp_int limit);
+extern mp_int get_memory_limit(short what);
+
 #endif /* XALLOC_H__ */
Index: src/lex.c
===================================================================
--- src/lex.c	(Revision 2505)
+++ src/lex.c	(Arbeitskopie)
@@ -804,7 +804,7 @@
     sprintf(mtext, "%d", ERQ_MAX_REPLY);
     add_permanent_define("__ERQ_MAX_REPLY__", -1, string_copy(mtext), MY_FALSE);
 #endif
-    sprintf(mtext, "%"PRIdMPINT, max_malloced);
+    sprintf(mtext, "%"PRIdMPINT, get_memory_limit(MALLOC_HARD_LIMIT));
     add_permanent_define("__MAX_MALLOC__", -1, string_copy(mtext), MY_FALSE);
     sprintf(mtext, "%"PRId32, def_eval_cost);
     add_permanent_define("__MAX_EVAL_COST__", -1, string_copy(mtext), MY_FALSE);
Index: src/slaballoc.c
===================================================================
--- src/slaballoc.c	(Revision 2505)
+++ src/slaballoc.c	(Arbeitskopie)
@@ -3254,7 +3254,7 @@
                 )
                )
             {
-                static const char mess[] = "MAX_MALLOCED limit reached.\n";
+                static const char mess[] = "HARD_MALLOC_LIMIT reached.\n";
                 writes(2, mess);
 
                 ptr = NULL;
Index: src/autoconf/configure.in
===================================================================
--- src/autoconf/configure.in	(Revision 2505)
+++ src/autoconf/configure.in	(Arbeitskopie)
@@ -217,7 +217,8 @@
 AC_MY_ARG_WITH(malloc,default,[default/smalloc/slaballoc/sysmalloc/ptmalloc],[memory manager to use])
 AC_MY_ARG_WITH(min-malloced,0,,)
 AC_MY_ARG_WITH(min-small-malloced,0,,)
-AC_MY_ARG_WITH(max-malloced,0x4000000,,)
+AC_MY_ARG_WITH(hard-malloc-limit,0x4000000,,)
+AC_MY_ARG_WITH(soft-malloc-limit,0,,)
 AC_MY_ARG_WITH(total-trace-length,0x1000,,)
 AC_MY_ARG_WITH(pcre-recursion-limit,3000,,[maximum number of recursions in PCRE package])
 AC_MY_ARG_WITH(wizlist-file,WIZLIST,,[name of the wizlist file])
@@ -462,7 +463,8 @@
 AC_INT_VAL_FROM_WITH(set_buffer_size_max)
 AC_INT_VAL_FROM_WITH(min_malloced)
 AC_INT_VAL_FROM_WITH(min_small_malloced)
-AC_INT_VAL_FROM_WITH(max_malloced)
+AC_INT_VAL_FROM_WITH(hard_malloc_limit)
+AC_INT_VAL_FROM_WITH(soft_malloc_limit)
 AC_INT_VAL_FROM_WITH(total_trace_length)
 AC_INT_VAL_FROM_WITH(pcre_recursion_limit)
 AC_INT_VAL_FROM_WITH(max_net_connects)
@@ -2820,7 +2822,8 @@
 AC_SUBST(val_malloc)
 AC_SUBST(val_min_malloced)
 AC_SUBST(val_min_small_malloced)
-AC_SUBST(val_max_malloced)
+AC_SUBST(val_hard_malloc_limit)
+AC_SUBST(val_soft_malloc_limit)
 AC_SUBST(val_total_trace_length)
 AC_SUBST(val_wizlist_file)
 AC_SUBST(val_pcre_recursion_limit)
Index: src/config.h.in
===================================================================
--- src/config.h.in	(Revision 2505)
+++ src/config.h.in	(Arbeitskopie)
@@ -137,10 +137,17 @@
 /* This value gives the upper limit for the total allocated memory
  * (useful for systems with no functioning process limit).
  * A value of 0 means 'unlimited'.
- * TODO: This should be named 'MAX_MALLOC' - ditto in configure.
  */
-#define MAX_MALLOCED         @val_max_malloced@
+#define HARD_MALLOC_LIMIT_DEFAULT    @val_hard_malloc_limit@
 
+/* This value gives a soft limit of the allocated memory (kind of low
+ * watermark). If this value is exceeded, the driver will call low_memory() in
+ * the master to inform the mudlib about the (potenntially) developing low
+ * memory situation.
+ * A value of 0 means 'unlimited'.
+ */
+#define SOFT_MALLOC_LIMIT_DEFAULT    @val_soft_malloc_limit@
+
 /* --- Random Number Generator (SFMT) --- */
 /* Set the period length of the SFMT.
  * Default is a period length of 2^19937 - 1
Index: src/backend.c
===================================================================
--- src/backend.c	(Revision 2505)
+++ src/backend.c	(Arbeitskopie)
@@ -490,6 +490,8 @@
 
         check_for_out_connections();
 
+        check_for_soft_malloc_limit();
+        
         if (prevent_object_cleanup)
         {
             if (num_listed_objs >= num_destructed/2)
@@ -541,6 +543,11 @@
                   write(1, buf, strlen(buf));
                   command_giver = NULL;
                   current_object = NULL;
+                  /* if the GC was not requested by an efun call, low_memory()
+                   * in the master is called to inform the game. */
+                  notify_lowmemory_condition(gc_request == gcEfun ?
+                                             NO_MALLOC_LIMIT_EXCEEDED : 
+                                             HARD_MALLOC_LIMIT_EXCEEDED);
                   garbage_collection();
                 }
                 else
Index: src/string_spec
===================================================================
--- src/string_spec	(Revision 2505)
+++ src/string_spec	(Arbeitskopie)
@@ -147,6 +147,7 @@
 INHERIT_FILE    "inherit_file"
 LOG_ERROR       "log_error"
 LOGON           "logon"
+LOW_MEMORY      "low_memory"
 PRELOAD         "preload"
 PREP_DEST       "prepare_destruct"
 PRINTF_OBJ_NAME "printf_obj_name"
Index: mudlib/sys/rtlimits.h
===================================================================
--- mudlib/sys/rtlimits.h	(Revision 2505)
+++ mudlib/sys/rtlimits.h	(Arbeitskopie)
@@ -21,4 +21,18 @@
 #define LIMIT_KEEP       (-1)  /* Keep the old limit setting */
 #define LIMIT_DEFAULT    (-2)  /* Use the default setting */
 
+/* memory limits */
+#define MALLOC_SOFT_LIMIT    1
+#define MALLOC_HARD_LIMIT    2
+
+/* low memory conditions */
+#define NO_MALLOC_LIMIT_EXCEEDED     0
+#define SOFT_MALLOC_LIMIT_EXCEEDED   MALLOC_SOFT_LIMIT
+#define HARD_MALLOC_LIMIT_EXCEEDED   MALLOC_HARD_LIMIT
+
+/* Flags for available reserves */
+#define USER_RESERVE_AVAILABLE     1
+#define MASTER_RESERVE_AVAILABLE   2
+#define SYSTEM_RESERVE_AVAILABLE   4
+
 #endif /* LPC_RTLIMITS_H_ */
low_memory.diff (33,505 bytes)   

Relationships

related to 0000601 resolvedzesstra Add possibility to change limits for memory allocation (MAX_MALLOCED, SOFT_MALLOC_LIMIT) at runtime 

Activities

zesstra

2009-01-08 07:06

administrator   ~0000869

I will have a look at this and check if it seems safe to for 3.3.719 or if we should postpone to 3.5. Right now, it doesn't seem to complicated.

Gnomi

2009-01-08 07:12

manager   ~0000870

Why should low_memory() return a value? The only thing it does is to prevent a GC run, is this such a good idea?

zesstra

2009-01-08 07:28

administrator   ~0000871

Im also still unsure, if it is a good idea to prevent it. But on the other hand, consider following (rather constructed) situation:
low_memory() is called and the mudlib decides a) that it would be better to perform the garbage_collection later because it is 19:00 and there are many players playing and b) it has some big objects and it can destruct them and free its memory. Then the mudlib would try to perform a GC during the next night when it disturbs fewer people.
But I think, in this case, the driver should make the GC anyway if the reserves could not be re-allocated (on the other hand: this could be even automatic: call low_memory() -> check if reserves can be re-allocated -> finish or do GC).

And taking this a step further:
It would be probably better to introduce the soft max-malloc (like a low watermark, while max-malloc would be the high watermark), call then low_memory() when hitting soft-max-malloc. The mudlib can then decide to initiate a GC at a convenient time in the near future. If max-malloc is hit before that time, the driver enforces the GC as usual.

Gnomi

2009-01-08 07:33

manager   ~0000873

I agree.

zesstra

2009-01-17 16:24

administrator   ~0000922

I used 'soft-malloc-limit' and SOFT_MALLOC_LIMIT. For the sake of consistency, I would actually suggest to rename max-malloced to hard-malloc-limit and MAX_MALLOCED to HARD_MALLOC_LIMIT... (Renaming the variable max_malloced is not that important, though no big effort.)

The patch is not quite ready, but the basic strategy is to check for the soft malloc limit once per backend cycle (not in every xalloc), that seems completely sufficient for me. If the limit is exceeded, low_memory() is called in the master once. After that another call is only done if the malloced memory shrinks below the soft limit (or the soft limit is raised accordingly).
Additionally I plan to call low_memory() as well, when the hard limit is reached, directly before the GC from the backend. That way, the mudlib master can inform users (the GC is not preventable).
Signature of low_memory(): void low_memory(int what, int limit, int memory_consumption);

zesstra

2009-01-18 10:58

administrator   ~0000928

Ok, first draft attached. Please comment. ;-)
I renamed the max-malloc to hard-malloc-limit and added the soft-malloc-limit. (The settings files of the various muds are already adapted.)
The soft limit is checked once per backend cycle and the low_memory() function is called as described in previous note.
I added get and set functions for the limits to ensure that the soft-limit is not >= hard-limit and that the hard-limit cannot be set lower then the already allocated memory, because that might lead to an immediate crash. This gets important once the limits can be changed at runtime.

The documentation is included.

An LPC side interface for querying and setting the limits is not in the patch. I will add them once this is finished.

zesstra

2009-02-28 16:14

administrator   ~0000962

Ok, patch for this is committed in r2523.

Issue History

Date Modified Username Field Change
2004-07-12 20:16 lars New Issue
2009-01-08 07:06 zesstra Note Added: 0000869
2009-01-08 07:06 zesstra Assigned To => zesstra
2009-01-08 07:06 zesstra Status new => assigned
2009-01-08 07:06 zesstra Product Version => 3.3
2009-01-08 07:06 zesstra Target Version => 3.3.719
2009-01-08 07:12 Gnomi Note Added: 0000870
2009-01-08 07:28 zesstra Note Added: 0000871
2009-01-08 07:33 Gnomi Note Added: 0000873
2009-01-17 12:07 zesstra Relationship added parent of 0000601
2009-01-17 16:24 zesstra Note Added: 0000922
2009-01-18 10:54 zesstra File Added: low_memory.diff
2009-01-18 10:58 zesstra Note Added: 0000928
2009-02-28 16:13 zesstra Relationship deleted parent of 0000601
2009-02-28 16:14 zesstra Relationship added related to 0000601
2009-02-28 16:14 zesstra Note Added: 0000962
2009-02-28 16:14 zesstra Status assigned => resolved
2009-02-28 16:14 zesstra Fixed in Version => 3.3.719
2009-02-28 16:14 zesstra Resolution open => fixed