View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0000081 | LDMud 3.3 | Runtime | public | 2004-07-12 20:16 | 2009-02-28 16:14 |
Reporter | Assigned To | zesstra | |||
Priority | normal | Severity | feature | Reproducibility | N/A |
Status | resolved | Resolution | fixed | ||
Product Version | 3.3 | ||||
Target Version | 3.3.719 | Fixed in Version | 3.3.719 | ||
Summary | 0000081: Better feedback on low memory situations. | ||||
Description | When the user/master/system reserve is freed (or reallocated), the master lfun low_memory() should be called, given as argument which reserves are currently unavailable. If the function returns 0, the standard low-memory handling (gc + reallocation attempt) is done. If the function returns 1, the driver assumes that the mudlib took care of things and will just attempt to reallocate the memory. In addition, a soft max-malloc limit is required - again, if it is exceeded, the low_memory() lfun is called. | ||||
Tags | No tags attached. | ||||
Attached Files | low_memory.diff (33,505 bytes)
Index: doc/driver/invocation =================================================================== --- doc/driver/invocation (Revision 2505) +++ doc/driver/invocation (Arbeitskopie) @@ -183,9 +183,16 @@ Giving this option results in smaller, but also more fragmented swapfiles, and the swap performance may degrade. - --max-malloc <size> - Restrict total memory allocations to <size> bytes. + --hard-malloc-limit <size> + Restrict total memory allocation to <size> bytes. A <size> of 0 or 'unlimited' removes any restriction.\n" + + --soft-malloc-limit <size> + This value gives a soft limit of the allocated memory (kind of low + watermark). If this value is exceeded, the driver will call low_memory() + in the master to inform the mudlib about the (potenntially) developing low + memory situation. + A <size> of 0 or 'unlimited' removes this threshold. --min-malloc <size> --min-small-malloc <size> Index: doc/master/low_memory =================================================================== --- doc/master/low_memory (Revision 0) +++ doc/master/low_memory (Revision 0) @@ -0,0 +1,58 @@ +SYNOPSIS + void low_memory(int what, int limitvalue, int memory, int reservestate) + +DESCRIPTION + This efun is called when there is a (potential) low-memory situation. + Two different limits for the amount of allocated memory can be + configured: a soft limit and a hard limit. + If the soft limit is exceeded, the driver calls low_memory() but does + nothing else. + If the hard limit is exceeded, the driver calls low_memory() as well, + but initiates a garbage collection directly after the call. + The efun is called as well directly before a user-initiated garbage + collection is started. + + <what> denotes the type of limit which was exceeded: + - NO_MALLOC_LIMIT_EXCEEDED (0) + No limit was exceeded, by a garbage collection was requested by a + call to garbage_collection() and it will begin right after + low_memory() returns. + - SOFT_MALLOC_LIMIT_EXCEEDED (1) + The soft limit was exceeded. + - HARD_MALLOC_LIMIT_EXCEEDED (2) + The hard limit was exceeded. A garbage collection will begin right + after low_memory() returns. + + <limit> specifies the numerical value of the limit which was exceeded. + It is 0 in case of NO_MALLOC_LIMIT_EXCEEDED. + + <memory> specifies the amount of allocated memory right now. + + <reservestate> specifies the current state of the memory reserves of + the driver. The states of the 3 different reserves are OR'ed together. + - USER_RESERVE_AVAILABLE (1) + The user reserve is available. + - MASTER_RESERVE_AVAILABLE (2) + The master reserve is available. + - SYSTEM_RESERVE_AVAILABLE (4) + The system reserve is available. + + This efun might be used to inform users about the garbage collection + and the expeced long lag. + Another possibility is to initiate a garbage collection deliberately + at a suitable time (e.g. during the following night) after the soft + limit was hit. + +REMARKS + If in a low_memory condition when the memory reserves have been used + already they could not be re-allocated after a garbage_collection() + slow_shut_down() is called instead of calling this function again. + +HISTORY + Introduced in 3.3.719 + +SEE ALSO + debug_info(E), garbage_collection(E) + slow_shut_down(M) + memory(C) + malloc(D) Index: doc/master/slow_shut_down =================================================================== --- doc/master/slow_shut_down (Revision 2505) +++ doc/master/slow_shut_down (Arbeitskopie) @@ -38,3 +38,4 @@ SEE ALSO quota_demon(M), notify_shutdown(M), shutdown(E), malloc(D), memory(C) + low_memory(M) Index: doc/concepts/memory =================================================================== --- doc/concepts/memory (Revision 2505) +++ doc/concepts/memory (Arbeitskopie) @@ -53,4 +53,6 @@ other objects. SEE ALSO - clean_up(A), slow_shut_down(M), quota_demon(M), malloc(D) + clean_up(A), slow_shut_down(M), quota_demon(M), low_memory(M) + malloc(D), + garbage_collection(E) Index: src/settings/lethynna =================================================================== --- src/settings/lethynna (Revision 2505) +++ src/settings/lethynna (Arbeitskopie) @@ -40,7 +40,7 @@ with_htable_size=65536 with_otable_size=8192 -with_max_malloced=0x20000000 +with_hard_malloc_limit=0x20000000 with_max_array_size=10000 with_max_mapping_keys=40000 Index: src/settings/evermore =================================================================== --- src/settings/evermore (Revision 2505) +++ src/settings/evermore (Arbeitskopie) @@ -96,7 +96,7 @@ with_reserved_user_size=800000 with_catch_reserved_cost=0xF000 with_master_reserved_cost=0xF000 -with_max_malloced=0 +with_hard_malloc_limit=0 enable_use_system_crypt=no enable_with_pthreads=no Index: src/settings/dt2 =================================================================== --- src/settings/dt2 (Revision 2505) +++ src/settings/dt2 (Arbeitskopie) @@ -32,7 +32,7 @@ with_max_players=100 with_min_malloced=0x1000000 -with_max_malloced=0x8000000 +with_hard_malloc_limit=0x8000000 with_htable_size=65536 with_itable_size=4096 Index: src/settings/gestrandet =================================================================== --- src/settings/gestrandet (Revision 2505) +++ src/settings/gestrandet (Arbeitskopie) @@ -23,7 +23,7 @@ with_reserved_user_size=0x400000 with_reserved_master_size=0x100000 with_reserved_system_size=0x200000 -with_max_malloced=0x1c000000 +with_hard_malloc_limit=0x1c000000 enable_min_malloced=yes with_min_malloced=0x8000000 enable_min_small_malloced=yes Index: src/settings/eotl =================================================================== --- src/settings/eotl (Revision 2505) +++ src/settings/eotl (Arbeitskopie) @@ -15,7 +15,7 @@ with_read_file_max_size=3000000 with_max_byte_transfer=500000 -with_max_malloced=0x6000000 +with_hard_malloc_limit=0x6000000 with_min_malloced=0x2000000 with_time_to_clean_up=7200 Index: src/settings/bat =================================================================== --- src/settings/bat (Revision 2505) +++ src/settings/bat (Arbeitskopie) @@ -65,7 +65,7 @@ with_compiler_stack_size=600 with_max_user_trace=120 with_max_trace=125 -with_max_malloced=0 +with_hard_malloc_limit=0 with_max_cost=20000000 with_catch_reserved_cost=150000 Index: src/settings/silberland =================================================================== --- src/settings/silberland (Revision 2505) +++ src/settings/silberland (Arbeitskopie) @@ -38,4 +38,4 @@ enable_use_parse_command=no enable_share_variables=no -with_max_malloced=0x5000000 +with_hard_malloc_limit=0x5000000 Index: src/settings/beutelland =================================================================== --- src/settings/beutelland (Revision 2505) +++ src/settings/beutelland (Arbeitskopie) @@ -18,7 +18,7 @@ with_reserved_user_size=0x800000 with_reserved_master_size=0x50000 with_reserved_system_size=0x100000 -with_max_malloced=0x16000000 +with_hard_malloc_limit=0x16000000 # --- Interpreter --- with_max_cost=1000000 Index: src/settings/dragonfire =================================================================== --- src/settings/dragonfire (Revision 2505) +++ src/settings/dragonfire (Arbeitskopie) @@ -41,7 +41,7 @@ --with-portno=1999 \ --with-udp-port=2000 \ --with-max-cost=2100000 \ ---with-max-malloced=0 \ +--with-hard-malloc-limit=0 \ --with-min-malloced=0x80000 \ --with-min-small-malloced=0x80000 \ --with-otable-size=8192 \ Index: src/settings/avalon =================================================================== --- src/settings/avalon (Revision 2505) +++ src/settings/avalon (Arbeitskopie) @@ -40,7 +40,7 @@ with_htable_size=65536 with_otable_size=8192 -with_max_malloced=0x23000000 +with_hard_malloc_limit=0x23000000 with_max_array_size=10000 with_max_mapping_keys=40000 Index: src/settings/pkmud =================================================================== --- src/settings/pkmud (Revision 2505) +++ src/settings/pkmud (Arbeitskopie) @@ -46,5 +46,5 @@ enable_share_variables=no with_min_malloced=0x80000 -with_max_malloced=0x5000000 +with_hard_malloc_limit=0x5000000 enable_dynamic_costs=yes Index: src/settings/default =================================================================== --- src/settings/default (Revision 2505) +++ src/settings/default (Arbeitskopie) @@ -127,7 +127,7 @@ # This value gives the upper limits for the total allocated memory. # A value of 0 means 'unlimited'. -with_max_malloced=0x4000000 +with_hard_malloc_limit=0x4000000 # If these values are >0, the driver will reserve this amount of memory # on startup for large resp. small blocks, reducing the block fragmentation. Index: src/settings/traumland =================================================================== --- src/settings/traumland (Revision 2505) +++ src/settings/traumland (Arbeitskopie) @@ -31,7 +31,7 @@ with_reserved_user_size=0x1000000 with_reserved_master_size=0x100000 with_reserved_system_size=0x400000 -with_max_malloced=0x10000000 +with_hard_malloc_limit=0x10000000 with_min_malloced=0x1000000 with_min_small_malloced=0x400000 Index: src/settings/unitopia =================================================================== --- src/settings/unitopia (Revision 2505) +++ src/settings/unitopia (Arbeitskopie) @@ -30,7 +30,7 @@ with_reserved_user_size=0x400000 with_reserved_master_size=0x100000 with_reserved_system_size=0x200000 -with_max_malloced=0x60000000 +with_hard_malloc_limit=0x60000000 enable_min_malloced=yes with_min_malloced=0x8000000 enable_min_small_malloced=yes Index: src/settings/morgengrauen =================================================================== --- src/settings/morgengrauen (Revision 2505) +++ src/settings/morgengrauen (Arbeitskopie) @@ -59,7 +59,7 @@ with_max_byte_transfer=200000 with_min_malloced=0xfa00000 with_min_small_malloced=0x8200000 -with_max_malloced=0x7fffffff +with_hard_malloc_limit=0x7fffffff with_pcre_recursion_limit=20000 with_optimize=med Index: src/settings/tubmud =================================================================== --- src/settings/tubmud (Revision 2505) +++ src/settings/tubmud (Arbeitskopie) @@ -31,7 +31,7 @@ # --- Memory --- with_reserved_user_size=0x100000 -with_max_malloced=0x06000000 +with_hard_malloc_limit=0x06000000 # --- Interpreter --- enable_strict_euids=no Index: src/settings/finalfrontier =================================================================== --- src/settings/finalfrontier (Revision 2505) +++ src/settings/finalfrontier (Arbeitskopie) @@ -32,7 +32,7 @@ with_reserved_user_size=0x400000 with_reserved_master_size=0x100000 with_reserved_system_size=0x200000 -with_max_malloced=0x16000000 +with_hard_malloc_limit=0x16000000 enable_min_malloced=yes with_min_malloced=0x8000000 enable_min_small_malloced=yes Index: src/settings/wunderland =================================================================== --- src/settings/wunderland (Revision 2505) +++ src/settings/wunderland (Arbeitskopie) @@ -106,7 +106,7 @@ --with-malloc=default \ --with-min-malloced=0 \ --with-min-small-malloced=0 \ ---with-max-malloced=0x6000000 \ +--with-hard-malloc-limit=0x6000000 \ --with-total-trace-length=0x1000 \ --with-wizlist-file=WIZLIST \ $* Index: src/smalloc.c =================================================================== --- src/smalloc.c (Revision 2505) +++ src/smalloc.c (Arbeitskopie) @@ -3039,7 +3039,7 @@ ) ) { - static const char mess[] = "MAX_MALLOCED limit reached.\n"; + static const char mess[] = "HARD_MALLOC_LIMIT reached.\n"; writes(2, mess); ptr = NULL; Index: src/xalloc.c =================================================================== --- src/xalloc.c (Revision 2505) +++ src/xalloc.c (Arbeitskopie) @@ -126,10 +126,19 @@ mp_int reserved_master_size = RESERVED_MASTER_SIZE; mp_int reserved_system_size = RESERVED_SYSTEM_SIZE; -mp_int min_malloced = MIN_MALLOCED; /* Allocation limits */ -mp_int min_small_malloced = MIN_SMALL_MALLOCED; /* Allocation limits */ -mp_int max_malloced = MAX_MALLOCED; +/* at startup reserve these amounts of memory for large and small blocks */ +mp_int min_malloced = MIN_MALLOCED; +mp_int min_small_malloced = MIN_SMALL_MALLOCED; +/* this is the hard limit for memory allocations. */ +static mp_int max_malloced = HARD_MALLOC_LIMIT_DEFAULT; +/* this is a soft limit for memory allocations. It serves as a kind of low + * watermark. If exceeded, the game driver will inform the mudlib by calling + * low_memory() in the master. */ +static mp_int soft_malloc_limit = SOFT_MALLOC_LIMIT_DEFAULT; +/* Was the low_memory() already called in the master`*/ +static Bool low_memory_applied = FALSE; + int stack_direction = 0; /* 0: Unknown stack behaviour * +1: Stack grows upward * -1: Stack grows downward @@ -180,7 +189,7 @@ /* Forward declarations */ #ifdef MALLOC_LPC_TRACE -static void write_lpc_trace (int d, word_t *p, int oneline); +static void write_lpc_trace (int d, word_t *p, int oneline) __attribute__((nonnull(2))); #endif #ifdef GC_SUPPORT @@ -518,7 +527,7 @@ #ifndef NO_MEM_BLOCK_SIZE if (max_malloced > 0 && (mp_int)xalloc_stat.size > max_malloced) { - static const char mess[] = "MAX_MALLOCED limit reached.\n"; + static const char mess[] = "HARD_MALLOC_LIMIT reached.\n"; writes(2, mess); if (malloc_privilege < MALLOC_SYSTEM) return MY_TRUE; @@ -1663,4 +1672,126 @@ return p; } /* string_copy_traced() */ +/*-------------------------------------------------------------------------*/ +void +notify_lowmemory_condition(short what) +/* Calls low_memory(what, <limit>, <memory_consumption>, <reserves> ) + * in the master. + */ +{ + short reservestate = 0; + push_number(inter_sp, what); + if (what == SOFT_MALLOC_LIMIT_EXCEEDED) + push_number(inter_sp, soft_malloc_limit); + else if (what == HARD_MALLOC_LIMIT_EXCEEDED) + push_number(inter_sp, max_malloced); + else + push_number(inter_sp, 0); + push_number(inter_sp, xalloc_stat.size); + if (reserved_user_area) + reservestate |= USER_RESERVE_AVAILABLE; + if (reserved_master_area) + reservestate |= MASTER_RESERVE_AVAILABLE; + if (reserved_system_area) + reservestate |= SYSTEM_RESERVE_AVAILABLE; + push_number(inter_sp, reservestate); + callback_master(STR_LOW_MEMORY, 4); +} /* inform_lowmemory_condition */ + +/*-------------------------------------------------------------------------*/ +void +check_for_soft_malloc_limit (void) +/* If soft_malloc_limit is set, check if the allocated memory exceeds it. + * If yes and the master was not notified until now, low_memory() is called + * in the mudlib master and low_memory_applied ist set to prevent any further + * notifications. + * If the limit is not exceeded, low_memory_applied is reset. + * Should be called from the backend. + */ +{ +#ifndef NO_MEM_BLOCK_SIZE + if (soft_malloc_limit > 0) + { + if ((mp_int)xalloc_stat.size > soft_malloc_limit) + { + if (!low_memory_applied) + { + /* call low_memory(malloced_memory) in the master but first + * set the flag to prevent calling every backend cycle in case + * of errors. */ + low_memory_applied = TRUE; + notify_lowmemory_condition(SOFT_MALLOC_LIMIT_EXCEEDED); + } + } + else if (low_memory_applied) + { + /* OK, memory consumption shrunk below the soft limit. Reset the + * warning, so that the next time the limit is exceeded, + * the master apply is done again. */ + low_memory_applied = FALSE; + } + } +#endif +} /* check_for_soft_malloc_limit() */ + +/*-------------------------------------------------------------------------*/ +Bool +set_memory_limit(short what, mp_int limit) +/* Sets the <what> memory limit to <limit>. + * <limit> for SOFT_MLIMIT has to be < HARD_MLIMIT. + * return TRUE on success and FALSE otherwise. + * If the current memory allocation is smaller than a new soft limit, the flag + * low_memory_applied is reset. + */ +{ +#ifndef NO_MEM_BLOCK_SIZE + // limits smaller than the sum of reserves (and some more) are harmful and + // lead anyway to immediate crashes... But we ignore this here, because + // reserve_memory() will deal with the needed sizes for the reserves and the + // minimum allocations. And later on in the game we will just prevent to + // set the limit smaller then the already allocated memory. + + if (what == MALLOC_SOFT_LIMIT) + { + if (limit >= max_malloced || limit < 0) + return FALSE; + + soft_malloc_limit = limit; + /* reset flag if appropriate. */ + if ((mp_int)xalloc_stat.size < soft_malloc_limit + && low_memory_applied) + low_memory_applied = FALSE; + } + else + { + // setting the limit below the currently allocated memory seems to be + // a very bad idea. + if (limit && limit <= (mp_int)xalloc_stat.size) + return FALSE; + + max_malloced = limit; + /* TODO: or disable it altogether? */ + if (soft_malloc_limit > max_malloced) + soft_malloc_limit = max_malloced - 1 ; + } + return TRUE; +#else + return FALSE; +#endif // NO_MEM_BLOCK_SIZE +} /* set_memory_limit */ + +/*-------------------------------------------------------------------------*/ +mp_int +get_memory_limit(short what) +/* Return the value of limit <what>. */ +{ +#ifndef NO_MEM_BLOCK_SIZE + if (what == MALLOC_SOFT_LIMIT) + return soft_malloc_limit; + else + return max_malloced; +#else + return 0; +#endif +} /***************************************************************************/ Index: src/main.c =================================================================== --- src/main.c (Revision 2505) +++ src/main.c (Arbeitskopie) @@ -1040,7 +1040,8 @@ , cHBInterval /* --heart-interval */ , cHostname /* --hostname */ , cHostaddr /* --hostaddr */ - , cMaxMalloc /* --max-malloc */ + , cMaxMalloc /* --hard-malloc-limit */ + , cSoftMallocLimit /* --soft-malloc-limit */ , cMaxArray /* --max-array */ , cMaxBytes /* --max-bytes */ , cMaxCallouts /* --max-callouts */ @@ -1364,13 +1365,22 @@ " Reuse free space in the swap file immediately.\n" } - , { 0, "max-malloc", cMaxMalloc, MY_TRUE - , " --max-malloc <size>\n" - , " --max-malloc <size>\n" - " Restrict total memory allocations to <size> bytes. A <size> of 0\n" + , { 0, "hard-malloc-limit", cMaxMalloc, MY_TRUE + , " --hard-malloc-lmit <size>\n" + , " --hard-malloc-limit <size>\n" + " Restrict total memory allocation to <size> bytes. A <size> of 0\n" " or 'unlimited' removes any restriction.\n" } + , { 0, "soft-malloc-limit", cSoftMallocLimit, MY_TRUE + , " --soft-malloc-lmit <size>\n" + , " --soft-malloc-limit <size>\n" + " If total memory allocation exceeds <size> bytes, inform the mudlib\n" + " master about a developing low memory situation. A <size> of 0\n" + " or 'unlimited' removes the threshold. <size> must be smaller than\n" + " --hard-malloc-limit.\n" + } + , { 0, "min-malloc", cMinMalloc, MY_TRUE , " --min-malloc <size>\n" , " --min-malloc <size>\n" @@ -1972,12 +1982,18 @@ , MIN_SMALL_MALLOCED ); - printf(" max allocation: "); - if (MAX_MALLOCED > 0) - printf("%9d\n", MAX_MALLOCED); + printf(" hard memory allocation limit: "); + if (HARD_MALLOC_LIMIT_DEFAULT > 0) + printf("%9d\n", HARD_MALLOC_LIMIT_DEFAULT); else printf("unlimited\n"); - + + printf(" soft memory allocation limit: "); + if (SOFT_MALLOC_LIMIT_DEFAULT > 0) + printf("%9d\n", SOFT_MALLOC_LIMIT_DEFAULT); + else + printf("unlimited\n"); + printf("Internal tables: shared string hash: %6d entries\n" " object hash: %6d entries\n" " reserved name hash: %6d entries\n" @@ -2513,19 +2529,33 @@ case cMaxMalloc: if (!strcasecmp(pValue, "unlimited")) { - max_malloced = 0; + set_memory_limit(MALLOC_HARD_LIMIT, 0); } else { - max_malloced = strtol(pValue, (char **)0, 0); - if (max_malloced < 0) + if (!set_memory_limit(MALLOC_HARD_LIMIT, strtol(pValue, (char **)0, 0))) { - fprintf(stderr, "Illegal value '%s' for --max-malloc\n", pValue); + fprintf(stderr, "Illegal value '%s' for --hard-malloc-limit\n", pValue); return hrError; } } break; + case cSoftMallocLimit: + if (!strcasecmp(pValue, "unlimited")) + { + set_memory_limit(MALLOC_SOFT_LIMIT, 0); + } + else + { + if (!set_memory_limit(MALLOC_SOFT_LIMIT, strtol(pValue, (char **)0, 0))) + { + fprintf(stderr, "Illegal value '%s' for --soft-malloc-limit\n", pValue); + return hrError; + } + } + break; + case cMudlib: if (chdir(pValue) == -1) { fprintf(stderr, "Bad mudlib directory: %s\n", pValue); Index: src/xalloc.h =================================================================== --- src/xalloc.h (Revision 2505) +++ src/xalloc.h (Arbeitskopie) @@ -67,10 +67,29 @@ /* --- Constants --- */ /* Allocation privilege levels */ +enum malloc_privileges { + MALLOC_USER = 0, + MALLOC_MASTER = 1, + MALLOC_SYSTEM = 2, +}; -#define MALLOC_USER (0) -#define MALLOC_MASTER (1) -#define MALLOC_SYSTEM (2) +/* memory limits */ +enum memory_limit_types { + MALLOC_SOFT_LIMIT = 1, + MALLOC_HARD_LIMIT = 2, +}; +/* low memory conditions */ +enum low_memory_conditions { + NO_MALLOC_LIMIT_EXCEEDED = 0, + SOFT_MALLOC_LIMIT_EXCEEDED = MALLOC_SOFT_LIMIT, + HARD_MALLOC_LIMIT_EXCEEDED = MALLOC_HARD_LIMIT, +}; +/* Flags for available reserves */ +enum available_reserve_flags { + USER_RESERVE_AVAILABLE = 1, + MASTER_RESERVE_AVAILABLE = 2, + SYSTEM_RESERVE_AVAILABLE = 4, +}; /* --- Variables --- */ @@ -84,7 +103,6 @@ extern mp_int reserved_system_size; extern mp_int min_malloced; extern mp_int min_small_malloced; -extern mp_int max_malloced; extern int stack_direction; @@ -122,43 +140,49 @@ #define xalloc_pass(size) xalloc_traced((size) MTRACE_PASS) #define rexalloc_pass(old,size) rexalloc_traced((old),(size) MTRACE_PASS) -extern size_t xalloced_size (POINTER p); +extern size_t xalloced_size (POINTER p) __attribute__((nonnull(1))); extern size_t xalloc_overhead (void); -extern POINTER xalloc_traced(size_t size MTRACE_DECL) - MALLOC __attribute__ ((warn_unused_result)); + +extern POINTER xalloc_traced(size_t size MTRACE_DECL) + MALLOC __attribute__ ((warn_unused_result)); + extern void xfree(POINTER); -extern POINTER rexalloc_traced(POINTER, size_t MTRACE_DECL) - MALLOC __attribute__ ((warn_unused_result)); -extern POINTER pxalloc_traced(size_t MTRACE_DECL) - MALLOC __attribute__ ((warn_unused_result)); -extern POINTER prexalloc_traced(POINTER, size_t MTRACE_DECL) - MALLOC __attribute__ ((warn_unused_result)); + +extern POINTER rexalloc_traced(POINTER, size_t size MTRACE_DECL) + MALLOC __attribute__((warn_unused_result)); + +extern POINTER pxalloc_traced(size_t size MTRACE_DECL) + MALLOC __attribute__ ((warn_unused_result)); +extern POINTER prexalloc_traced(POINTER, size_t size MTRACE_DECL) + MALLOC __attribute__((warn_unused_result)); + extern void pfree(POINTER); -extern void * malloc_increment_size (void *vp, size_t size); +extern void * malloc_increment_size (void *vp, size_t size) + __attribute__((nonnull(1))); #ifdef GC_SUPPORT -extern void x_clear_ref (POINTER p); -extern int x_mark_ref (POINTER p); -extern Bool x_test_ref (POINTER p); +extern void x_clear_ref (POINTER p) __attribute__((nonnull(1))); +extern int x_mark_ref (POINTER p) __attribute__((nonnull(1))); +extern Bool x_test_ref (POINTER p) __attribute__((nonnull(1))); #endif /* GC_SUPPORT */ #ifdef MALLOC_TRACE extern void store_print_block_dispatch_info(void *block, void (*func)(int, void *, int) ); -extern Bool is_freed(void *p, p_uint minsize); +extern Bool is_freed(void *p, p_uint minsize) __attribute__((nonnull(1))); #endif /* MALLOC_TRACE */ #ifdef CHECK_OBJECT_GC_REF -extern void note_object_allocation_info ( void *block ); -extern void note_program_allocation_info ( void *block ); -extern Bool is_object_allocation ( void *block ); -extern Bool is_program_allocation ( void *block ); +extern void note_object_allocation_info ( void *block ) __attribute__((nonnull(1))); +extern void note_program_allocation_info ( void *block ) __attribute__((nonnull(1))); +extern Bool is_object_allocation ( void *block ) __attribute__((nonnull(1))); +extern Bool is_program_allocation ( void *block ) __attribute__((nonnull(1))); #endif /* CHECK_OBJECT_RC_REF */ /* Functions directly exported from the allocator: */ -extern void mem_dump_data(strbuf_t *sbuf); -extern void mem_dump_extdata(strbuf_t *sbuf); -extern void mem_dinfo_data(svalue_t *svp, int value); +extern void mem_dump_data(strbuf_t *sbuf) __attribute__((nonnull(1))); +extern void mem_dump_extdata(strbuf_t *sbuf) __attribute__((nonnull(1))); +extern void mem_dinfo_data(svalue_t *svp, int value) __attribute__((nonnull(1))); extern void mem_consolidate (Bool force); extern Bool mem_dump_memory(int fd); #ifdef MALLOC_EXT_STATISTICS @@ -178,13 +202,19 @@ #define string_copy(s) string_copy_traced(s) #endif -extern char * string_copy_traced(const char *str MTRACE_DECL) MALLOC; -extern void dump_lpc_trace (int d, void *p); -extern void dump_malloc_trace (int d, void *adr); +extern char * string_copy_traced(const char *str MTRACE_DECL) MALLOC + __attribute__((nonnull(1))) __attribute__((warn_unused_result)); +extern void dump_lpc_trace (int d, void *p) __attribute__((nonnull(2))); +extern void dump_malloc_trace (int d, void *adr) __attribute__((nonnull(2))); extern void get_stack_direction (void); extern void assert_stack_gap(void); extern void reserve_memory (void); extern void reallocate_reserved_areas(void); +extern void check_for_soft_malloc_limit(void); +extern void notify_lowmemory_condition(short what); +extern Bool set_memory_limit(short what, mp_int limit); +extern mp_int get_memory_limit(short what); + #endif /* XALLOC_H__ */ Index: src/lex.c =================================================================== --- src/lex.c (Revision 2505) +++ src/lex.c (Arbeitskopie) @@ -804,7 +804,7 @@ sprintf(mtext, "%d", ERQ_MAX_REPLY); add_permanent_define("__ERQ_MAX_REPLY__", -1, string_copy(mtext), MY_FALSE); #endif - sprintf(mtext, "%"PRIdMPINT, max_malloced); + sprintf(mtext, "%"PRIdMPINT, get_memory_limit(MALLOC_HARD_LIMIT)); add_permanent_define("__MAX_MALLOC__", -1, string_copy(mtext), MY_FALSE); sprintf(mtext, "%"PRId32, def_eval_cost); add_permanent_define("__MAX_EVAL_COST__", -1, string_copy(mtext), MY_FALSE); Index: src/slaballoc.c =================================================================== --- src/slaballoc.c (Revision 2505) +++ src/slaballoc.c (Arbeitskopie) @@ -3254,7 +3254,7 @@ ) ) { - static const char mess[] = "MAX_MALLOCED limit reached.\n"; + static const char mess[] = "HARD_MALLOC_LIMIT reached.\n"; writes(2, mess); ptr = NULL; Index: src/autoconf/configure.in =================================================================== --- src/autoconf/configure.in (Revision 2505) +++ src/autoconf/configure.in (Arbeitskopie) @@ -217,7 +217,8 @@ AC_MY_ARG_WITH(malloc,default,[default/smalloc/slaballoc/sysmalloc/ptmalloc],[memory manager to use]) AC_MY_ARG_WITH(min-malloced,0,,) AC_MY_ARG_WITH(min-small-malloced,0,,) -AC_MY_ARG_WITH(max-malloced,0x4000000,,) +AC_MY_ARG_WITH(hard-malloc-limit,0x4000000,,) +AC_MY_ARG_WITH(soft-malloc-limit,0,,) AC_MY_ARG_WITH(total-trace-length,0x1000,,) AC_MY_ARG_WITH(pcre-recursion-limit,3000,,[maximum number of recursions in PCRE package]) AC_MY_ARG_WITH(wizlist-file,WIZLIST,,[name of the wizlist file]) @@ -462,7 +463,8 @@ AC_INT_VAL_FROM_WITH(set_buffer_size_max) AC_INT_VAL_FROM_WITH(min_malloced) AC_INT_VAL_FROM_WITH(min_small_malloced) -AC_INT_VAL_FROM_WITH(max_malloced) +AC_INT_VAL_FROM_WITH(hard_malloc_limit) +AC_INT_VAL_FROM_WITH(soft_malloc_limit) AC_INT_VAL_FROM_WITH(total_trace_length) AC_INT_VAL_FROM_WITH(pcre_recursion_limit) AC_INT_VAL_FROM_WITH(max_net_connects) @@ -2820,7 +2822,8 @@ AC_SUBST(val_malloc) AC_SUBST(val_min_malloced) AC_SUBST(val_min_small_malloced) -AC_SUBST(val_max_malloced) +AC_SUBST(val_hard_malloc_limit) +AC_SUBST(val_soft_malloc_limit) AC_SUBST(val_total_trace_length) AC_SUBST(val_wizlist_file) AC_SUBST(val_pcre_recursion_limit) Index: src/config.h.in =================================================================== --- src/config.h.in (Revision 2505) +++ src/config.h.in (Arbeitskopie) @@ -137,10 +137,17 @@ /* This value gives the upper limit for the total allocated memory * (useful for systems with no functioning process limit). * A value of 0 means 'unlimited'. - * TODO: This should be named 'MAX_MALLOC' - ditto in configure. */ -#define MAX_MALLOCED @val_max_malloced@ +#define HARD_MALLOC_LIMIT_DEFAULT @val_hard_malloc_limit@ +/* This value gives a soft limit of the allocated memory (kind of low + * watermark). If this value is exceeded, the driver will call low_memory() in + * the master to inform the mudlib about the (potenntially) developing low + * memory situation. + * A value of 0 means 'unlimited'. + */ +#define SOFT_MALLOC_LIMIT_DEFAULT @val_soft_malloc_limit@ + /* --- Random Number Generator (SFMT) --- */ /* Set the period length of the SFMT. * Default is a period length of 2^19937 - 1 Index: src/backend.c =================================================================== --- src/backend.c (Revision 2505) +++ src/backend.c (Arbeitskopie) @@ -490,6 +490,8 @@ check_for_out_connections(); + check_for_soft_malloc_limit(); + if (prevent_object_cleanup) { if (num_listed_objs >= num_destructed/2) @@ -541,6 +543,11 @@ write(1, buf, strlen(buf)); command_giver = NULL; current_object = NULL; + /* if the GC was not requested by an efun call, low_memory() + * in the master is called to inform the game. */ + notify_lowmemory_condition(gc_request == gcEfun ? + NO_MALLOC_LIMIT_EXCEEDED : + HARD_MALLOC_LIMIT_EXCEEDED); garbage_collection(); } else Index: src/string_spec =================================================================== --- src/string_spec (Revision 2505) +++ src/string_spec (Arbeitskopie) @@ -147,6 +147,7 @@ INHERIT_FILE "inherit_file" LOG_ERROR "log_error" LOGON "logon" +LOW_MEMORY "low_memory" PRELOAD "preload" PREP_DEST "prepare_destruct" PRINTF_OBJ_NAME "printf_obj_name" Index: mudlib/sys/rtlimits.h =================================================================== --- mudlib/sys/rtlimits.h (Revision 2505) +++ mudlib/sys/rtlimits.h (Arbeitskopie) @@ -21,4 +21,18 @@ #define LIMIT_KEEP (-1) /* Keep the old limit setting */ #define LIMIT_DEFAULT (-2) /* Use the default setting */ +/* memory limits */ +#define MALLOC_SOFT_LIMIT 1 +#define MALLOC_HARD_LIMIT 2 + +/* low memory conditions */ +#define NO_MALLOC_LIMIT_EXCEEDED 0 +#define SOFT_MALLOC_LIMIT_EXCEEDED MALLOC_SOFT_LIMIT +#define HARD_MALLOC_LIMIT_EXCEEDED MALLOC_HARD_LIMIT + +/* Flags for available reserves */ +#define USER_RESERVE_AVAILABLE 1 +#define MASTER_RESERVE_AVAILABLE 2 +#define SYSTEM_RESERVE_AVAILABLE 4 + #endif /* LPC_RTLIMITS_H_ */ | ||||
|
I will have a look at this and check if it seems safe to for 3.3.719 or if we should postpone to 3.5. Right now, it doesn't seem to complicated. |
|
Why should low_memory() return a value? The only thing it does is to prevent a GC run, is this such a good idea? |
|
Im also still unsure, if it is a good idea to prevent it. But on the other hand, consider following (rather constructed) situation: low_memory() is called and the mudlib decides a) that it would be better to perform the garbage_collection later because it is 19:00 and there are many players playing and b) it has some big objects and it can destruct them and free its memory. Then the mudlib would try to perform a GC during the next night when it disturbs fewer people. But I think, in this case, the driver should make the GC anyway if the reserves could not be re-allocated (on the other hand: this could be even automatic: call low_memory() -> check if reserves can be re-allocated -> finish or do GC). And taking this a step further: It would be probably better to introduce the soft max-malloc (like a low watermark, while max-malloc would be the high watermark), call then low_memory() when hitting soft-max-malloc. The mudlib can then decide to initiate a GC at a convenient time in the near future. If max-malloc is hit before that time, the driver enforces the GC as usual. |
|
I agree. |
|
I used 'soft-malloc-limit' and SOFT_MALLOC_LIMIT. For the sake of consistency, I would actually suggest to rename max-malloced to hard-malloc-limit and MAX_MALLOCED to HARD_MALLOC_LIMIT... (Renaming the variable max_malloced is not that important, though no big effort.) The patch is not quite ready, but the basic strategy is to check for the soft malloc limit once per backend cycle (not in every xalloc), that seems completely sufficient for me. If the limit is exceeded, low_memory() is called in the master once. After that another call is only done if the malloced memory shrinks below the soft limit (or the soft limit is raised accordingly). Additionally I plan to call low_memory() as well, when the hard limit is reached, directly before the GC from the backend. That way, the mudlib master can inform users (the GC is not preventable). Signature of low_memory(): void low_memory(int what, int limit, int memory_consumption); |
|
Ok, first draft attached. Please comment. ;-) I renamed the max-malloc to hard-malloc-limit and added the soft-malloc-limit. (The settings files of the various muds are already adapted.) The soft limit is checked once per backend cycle and the low_memory() function is called as described in previous note. I added get and set functions for the limits to ensure that the soft-limit is not >= hard-limit and that the hard-limit cannot be set lower then the already allocated memory, because that might lead to an immediate crash. This gets important once the limits can be changed at runtime. The documentation is included. An LPC side interface for querying and setting the limits is not in the patch. I will add them once this is finished. |
|
Ok, patch for this is committed in r2523. |
Date Modified | Username | Field | Change |
---|---|---|---|
2004-07-12 20:16 |
|
New Issue | |
2009-01-08 07:06 | zesstra | Note Added: 0000869 | |
2009-01-08 07:06 | zesstra | Assigned To | => zesstra |
2009-01-08 07:06 | zesstra | Status | new => assigned |
2009-01-08 07:06 | zesstra | Product Version | => 3.3 |
2009-01-08 07:06 | zesstra | Target Version | => 3.3.719 |
2009-01-08 07:12 | Gnomi | Note Added: 0000870 | |
2009-01-08 07:28 | zesstra | Note Added: 0000871 | |
2009-01-08 07:33 | Gnomi | Note Added: 0000873 | |
2009-01-17 12:07 | zesstra | Relationship added | parent of 0000601 |
2009-01-17 16:24 | zesstra | Note Added: 0000922 | |
2009-01-18 10:54 | zesstra | File Added: low_memory.diff | |
2009-01-18 10:58 | zesstra | Note Added: 0000928 | |
2009-02-28 16:13 | zesstra | Relationship deleted | parent of 0000601 |
2009-02-28 16:14 | zesstra | Relationship added | related to 0000601 |
2009-02-28 16:14 | zesstra | Note Added: 0000962 | |
2009-02-28 16:14 | zesstra | Status | assigned => resolved |
2009-02-28 16:14 | zesstra | Fixed in Version | => 3.3.719 |
2009-02-28 16:14 | zesstra | Resolution | open => fixed |