Memory fragments

Published on May 1, 2006 By CariElf In GalCiv Journals

Our top priority since 1.1 came out has been resolving the memory and performance issues that people have been having. BoundsChecker didn't come up with any significant memory leaks, so that left us with checking the change logs to see what we might have done to cause the issues. We were obviously doing something wacky, because people with 4 GB page files were still getting out of virtual memory errors.

One of the changes we made was to the way that the shipcfg files were parsed. The shipcfg files are really just ini files, which I think Joe picked for the shipcfg format because ini files are supposed to be fast for reading and writing. They have two main disadvantages. The first is that you can't have an ini file greater than 64 kb on Windows ME and Windows 98; it won't be able to read anything past the 64 kb mark. The second is that you have to know what is the size of the longest section name, and the longest key name, or else know what all the section and key names are before you go to read in the data from the file. If you don't know the section and keynames, you have to call APIs to get all the section and keynames that are in the file, and you need a buffer to store them.

There are two ways to allocate memory: static and dynamic. Static means that you always allocate the same amount of memory, no matter how much of it you actually need to use. If you allocate too much, you'll be taking up memory you aren't using, but if you don't allocate enough, you'll run into problems because you don't have enough to use. Dynamic memory is allocated as you need it, when you need it. Since you're in charge of dynamic memory, you have to remember to deallocate (release) it. When you forget to deallocate it, it becomes a memory leak. As far as your program and OS are concerned, that memory is still being used and is unavailable to be allocated to something else. Another bad thing about dynamic allocation is that you can fragment memory.

I'm assuming that most of you have seen Windows' disk defrag program. If you haven't, you might want to go to Start->All Programs->Accessories->System Tools and run it, because your had drive probably needs it.

It will also give you a visual indication of what I'm talking about in this paragraph. When you create or copy files to your hard drive, the files are copied into consecutive blocks of memory. If you delete one or more of those files, that leaves a hole in memory. Depending on how big the hole is, it might get filled with other files. But if it's too small, it's just wasted. Disk defrag goes through your hard drive and tries to move stuff around so that it is stored more efficiently, with bigger blocks of available memory. Fragmentation can also happen in system memory, aka RAM. So even if you deallocate dynamic memory, there's a chance that the memory you released won't be used again by your program, so it keeps using more memory. When the program runs out of available RAM, it will start using virtual memory. Virtual memory is really just a section of the hard drive that is set aside for the operating system's use when it runs out of RAM.

So how does all of this relate to GC2's issues? The shipcfg files originally used static memory allocation for the buffer that was used to get the section and keynames, but if you added enough jewelry and components to the ship, the buffer wasn't big enough. We needed a quick way to fix this that wouldn't involve re-engineering the shipcfg code and that wouldn't make reading in the shipcfg files take longer. The quickest change to implement was to switch from using static allocation to using dynamic allocation. In order to make sure that the buffer was big enough, I suggested that we dynamically allocate a buffer that was the same size as the file. The bad thing about this solution was that it didn't take into account that we weren't saving the parsed data values in memory after reading in the file for the first time. Every time you built a colony ship, the colony ship cfg file was read in and a buffer was allocated and deallocated. So I'm thinking that we were probably fragmenting memory.

Last week, I wrote code to make GC2 saved the parsed values from the shipcfg files in memory, so that they would only have to be read in once. It would mean that we're hanging on to a little more memory than we were before, but it should cut down on the fragmentation. It definitely cuts down on the amount of time spent creating ship graphics, which you will notice when loading a save game. If it's not enough, we may still have to change the shipcfg file format, but keeping the parsed results in memory will help keep load times down. I wrote the new shipcfg code in such a way that I should only have to replace one function if we need to switch the file format, the one that actually reads in the data. The code that uses the stored data to put together a ship with all its components will not need to be changed.

Another problem area is the save game code. Writing to the hard disk is slow, so it's quicker to create the file in memory and then write it out to the file in one fell swoop rather then writing out each datum as you go. Since the exact file size of a given save game is unknown, the save game code uses dynamic memory allocation. Each object in the game (ie ships, planets, civs, etc) has its own function to create a block of memory containing all the data it needs to store, which it then passes to the main save game function, and is added to the main block. This is using the same code as in GC1, but we had less dynamic data in GC1. Originally, all of the buffers started out as 1 kb and whenever the buffers needed to increase in size, they would allocate their current size + 1 kb and copy the data from the old buffer to the new buffer, then deallocate the old buffer. The process of growing and copying the buffers was taking up more time than the actual saving of data, and I needed a way to improve performance without doing major surgery to the save game code before version 1.0 came out. So I did some profiling on how big the buffers were for a gigantic galaxy in the first few turns of the game and how much they grew, and used those numbers to change the initial buffer sizes and how much they grew by for each data object. This was, admittedly, more of a band-aid than an actual fix.

Apart from adding some new things that needed to be saved, I don't think that we've really touched the save game code much. However, it is still fairly inefficient because of all the buffer growing and copying. So the next change I have started to make is to make all the data objects use one buffer. Once that is done, I can make further optimizations to the code like initializing the buffer size based on the galaxy size, and see if it does more good than harm to keep the buffer in memory so that it doesn't have to allocate and deallocate 2-13 MB (or more) every time the game saves. At the very least, making everything use one buffer should cut down on fragmentation and make the saving go quicker. I will also be reviewing the code to make sure that only necessary data is being saved rather than being recalculated, in an effort to cut down loading time.

Once I've finished making our memory usage more efficient, I'll start working on the modding stuff again.

Edit: Ok, since I'm getting e-mails and comments about this, I would like to clarify that I am not blaming your hard drives for causing the crashes. The point of this article is that I am working on resolving the memory issues. The comment about running disk defrag was meant as a general statement that you should regularly defrag your hard drives, and to provide a visual representation of what is happening when memory is fragmented.

Update: In my sticky thread here Link I put instructions and a link for an unofficial test exe.
--Cari

Comments (Page 2)

whismerhill

on May 01, 2006

ok thanks for the answer Cari
I wasn't talking about bugs particular to anything one might have as software or hardware though
but rather some that affects everyone(at least as far as I know)

anyway I can understand

not feeling ignored is enough for me

I'll wait

ripsql

on May 01, 2006

CariElf and Lord Reliant,

The issues you mentioned is interesting...
I generally have the game running full screen but minimized while doing something else and come back etc.
Have it up several days, but occasionaly I would check on my computer and the game would not be running.
It is as if the game ended, no errors.
This happens about once 3 days of continuously running...alt-tab back and forth.
Had the game be closed when I get to my system about 3 to 4 times in two weeks.

I should check the debug file when this happens to check for the bug.
I got into the habit of quick saving before leaving the computer.
Never knew about that bug...

Moosetek13

on May 01, 2006

Well, I for one don't know what the heck everyone is complaining about. Saves (first, second, quick, you name it) take about a second. This version is the most stable I have seen so far. It is far more stable than other games I play, like Sim City 4. One little problem with Rangers coming out of the woodwork, which is probably on their back burner - understandably - is the only glitch I have seen so far.

I just hope the fixes to these (other peoples - hehe) problems don't introduce more to my own system.

S1Genocide

on May 01, 2006

Well at least they admitted theres a problem. It disturbs me how quickly someone gets flamed for posting a bug in bug reports.

And yes this version is stable but its laggy and slow compared to the last BETA.

jpinard

on May 01, 2006

I'd like to add some comments about the issues listed above since it appears a lot of people only come here to post "problems" - and it may appear on the outside that the majority of people have issues, which can't be true. Since the last beta patch and with v1.1 - the game on my slightly unstable - highly overclocked system is 100% stable, up from ~99.5% previously. I mention the stability not to say "yea for me!" - but because I've been working on some mods and therefore running a bunch of junk simulataneously (on a single core cpu) - iTunes for music, Paint Shop Pro 7.04, Gal Civ 2 of course, Ultredit, full suite of AV software, speedfan, and several monitors for my Logitech Gamer Keyboard (G15). The keyboard I have is an important utility because it allows me to see my page file usage, system RAM, cpu utilization on its own LCD screen. So I watch these figures constantly. Right now with iTunes in the background and this webpage on I'm using on average 3% cpu time and 28% of my system RAM (out of 2 Gig of RAM). When I launch Gal Civ 2, my cpu utililization rises to 100% just like it should, but my RAM only increases to ~46% with all the above programs running. With Gal Civ alone it rises to about 40%. Even after playing a long time, I don't see the RAM usage climb considerably (usually barely at all) indicating there's not an identifiable memory leak on my system. What I'm getting at is this could be system dependant which would make isolating some of the issues exceedingly difficult. Maybe it would be helpful if multiple people use the exact same ship and savegame file to see what kind of hit it has on differing systems... You may not think your motherboard could impact something like this, but the fact is - it can make a huge difference. When Asus released the A8R-MVP one of the first Crossfire boards in October 2005 it seemed to be "OK". However as the new line of Radeon video cards came out (x1900) and ATI optimized the drivers for the 512 Meg of video memory onboard - many of us witnessed a remarkable drop in cpu performance on the order of 35%. So while people with slower video cards & slower cpu's were experiencing increases with each new Catalyst driver release, many of saw the opposite. Indications first would be "it's ATI's fault" since their CAT 6.1 driver release first showed the hit. However... in the end working with ATI engineers (since Asus wasn't responsive), we isolated the issue to a single cpu register Asus had not programmed correctly in the A8R-MVP BIOS. All this in a motherboard that had already been out for 5 months. ATI had vertex buffers in their D3D driver to optimize the corresponding data stream. This optimization exposed a major flaw in the Asus BIOS... which was ONLY evident on a single board - the A8R-MVP - and only in some programs.

So in regards to the 4 Gig page file usage - considering I can't get the game too use more than 1 Gig of RAM while multitasking...that's just really odd. My pagefile commit charge is pretty steady at ~750-800 Meg. But I manually set pagefile is to a min/max value of 2048 Meg (2 Gig) so maybe a persons swap file setup is contributing to the issue...

Spleeze (Brian) - I ran into this as well, but restarting GC plus starting a fresh new game with the changes fixed the issue (when it ocurred). I'm surprised anything shows up during on-the-fly editing without a full restart of the renderer. Most games I mod require a full restart - but I've found I can play with a lot of things in GC, alt-tab back to the game and they're there.

Cari, have you run a poll of computer specs of those that purcahsed GC2? Like the one Valve ran?http://www.steampowered.com/status/survey.html
I'm surprised you'd have any concern with Win 98jpinard users since it seems the number of people using it for gaming would be exceedingly small? Also, anything wrong with static instead of dynamic allocation? If a person is nio technically inclined, they should have windows managing their pagefile and only modders would really be in need of multi-tasking benefist of dynamic mem use. And if a person is a modder, it seems they're be smart enough to adjust page file useage ior probably already have adequate RAM. Just curious bcause it sounds like static assignment would be a lot faster/easier to solve the issue listed above, than what you're looking at now?

And thanks (all of Stardock) for such excellent support.

spleeze_

on May 01, 2006

Hey Jeffrey Pinard.

I actually do perform a full game restart whenever I update my models. I even started the game twice, and the thumbs didn’t update. I wouldn’t be surprised if, after a fresh install of GC2, and installing the ships right away, everything would show up properly. Also, the ships will be released using the mod architecture, which I’m sure will better accommodate changes to core content.

Thanks for the info.

Oh, what kind of modding are you doing for GC2? Lemme know. galtrek@splicedigital.com

-Brian

Peace Phoenix

on May 02, 2006

So how does all of this relate to GC2's issues? The shipcfg files originally used static memory allocation for the buffer that was used to get the section and keynames, but if you added enough jewelry and components to the ship, the buffer wasn't big enough. We needed a quick way to fix this that wouldn't involve re-engineering the shipcfg code and that wouldn't make reading in the shipcfg files take longer. The quickest change to implement was to switch from using static allocation to using dynamic allocation. In order to make sure that the buffer was big enough, I suggested that we dynamically allocate a buffer that was the same size as the file. The bad thing about this solution was that it didn't take into account that we weren't saving the parsed data values in memory after reading in the file for the first time. Every time you built a colony ship, the colony ship cfg file was read in and a buffer was allocated and deallocated. So I'm thinking that we were probably fragmenting memory.

Does it work the same way the AI ships?

Each object in the game (ie ships, planets, civs, etc) has its own function to create a block of memory containing all the data it needs to store, which it then passes to the main save game function, and is added to the main block

I can make further optimizations to the code like initializing the buffer size based on the galaxy size, and see if it does more good than harm to keep the buffer in memory so that it doesn't have to allocate and deallocate 2-13 MB (or more) every time the game saves.

If I understand correctly, does this mean that you will keep in memory the contents of files corresponding to data wthat won't change. Unless I am mistaken, but once a ship has been designed, the associated shpcfg data won't change until the next use of the ship designer on that design. Maybe a dictionnary like structure maybe be useful for keeping track of buffers that you decide not to regenerate each time you are doing a save. Sometimes, it is more efficient to calculate some data one time for all and memorize them instead of calculating them each time, especially if they aren't subject to lots of modification.

jpinard

on May 02, 2006

Hi Brian. E-mailed you.

Mostly working on mods to help with visual identification at the moment.

What cracks me up, is this website causes me shutdown errors - not the game - hehe
-Jeff

alienrider

on May 02, 2006

Thanks for the detail. As a programmer it makes interesting reading.

EZ

AR

Alfb

on May 02, 2006

It looks like you don't understand what static and dynamic memory refers to. In a program, you have the ability to create variables which take up memory. Without getting into too much detail, you can create a variable two ways, statically and dynamically. The static way means that memory is allocated to the program and stays allocated until the program ends. Using this allows compilers to efficiently utilize space used but also means that that space is permanently used for the life of the program. Dynamically allocating memory is usually used when something is short term and can be destroyed and the memory given back to the system (this is one way for leaks to happen if you forget to deallocate). Lets say you have 10 variables used and to keep it simple they use 1 byte and are never needed at the same time. Statically, they will use 10 bytes for the entire time the program runs. Dynamically they will use 1 byte only when one is being used since there is only 1 of them in existence at any time. There are also issues about where static memory is versus dynamic is stored but Heaps etc are beyond this topic. In the end, they have to do with how memory is used and how much memory is used by a program and don't really have to do with virtual memory and how it is set up by users.

CariElf

on May 02, 2006

Also, anything wrong with static instead of dynamic allocation?

Well, if we use static allocation for the buffer that reads in the shipcfg files, you would be limited to how much jewelry you could put on ships, because if the section names and key names got longer than the buffer size, stuff would get cut off.

Does it work the same way the AI ships?

Well, the AI reads in their basic hull and configuration from a shipcfg file, but then they add their weapons, etc. So yes, it will also affect the AI ships.

The shipcfgs will now stay in memory, parsed, until a new game is created or a save game loaded. Then they will be cleared so that there aren't a ton of shipcfgs in memory that aren't being used.

I used to save all the definitions like techs, planet improvements, etc, in a file in the temp folder, and then just copy the data if it hadn't changed, but every time you design a new ship, it changes what needs to be in that block, so it was almost always re-saving that block anyway. I could probably do it for everything except the ship designs and then just save the ship designs every time.

Vulcannis

on May 02, 2006

Originally, all of the buffers started out as 1 kb and whenever the buffers needed to increase in size, they would allocate their current size + 1 kb and copy the data from the old buffer to the new buffer, then deallocate the old buffer.

Ouch, O(n^2). I hope you guys are now growing buffers by a factor rather than just incrementing them. Log for the win.

I'm surprised though that heap fragmentation is actually an issue... would it not be required that the allocations be both asymptotically increasing in size, and their deallocations be sporadic? Maybe do a profile to see if things are hanging around too long and try and deallocate them earlier if possible?

jpinard

on May 02, 2006

Also, anything wrong with static instead of dynamic allocation? If a person is nio technically inclined, they should have windows managing their pagefile and only modders would really be in need of multi-tasking benefist of dynamic mem use. And if a person is a modder, it seems they're be smart enough to adjust page file useage ior probably already have adequate RAM. Just curious bcause it sounds like static assignment would be a lot faster/easier to solve the issue listed above, than what you're looking at now?

It looks like you don't understand what static and dynamic memory refers to. In a program, you have the ability to create variables which take up memory. Without getting into too much detail, you can create a variable two ways, statically and dynamically. The static way means that memory is allocated to the program and stays allocated until the program ends. Using this allows compilers to efficiently utilize space used but also means that that space is permanently used for the life of the program. Dynamically allocating memory is usually used when something is short term and can be destroyed and the memory given back to the system (this is one way for leaks to happen if you forget to deallocate). Lets say you have 10 variables used and to keep it simple they use 1 byte and are never needed at the same time. Statically, they will use 10 bytes for the entire time the program runs. Dynamically they will use 1 byte only when one is being used since there is only 1 of them in existence at any time. There are also issues about where static memory is versus dynamic is stored but Heaps etc are beyond this topic. In the end, they have to do with how memory is used and how much memory is used by a program and don't really have to do with virtual memory and how it is set up by users.

I didn't write what I was thinking too well. I was referring to the fact people are reporting page file usage maxing out at 4+ Gig and Cari mentioning it was because dynamic memory allocation is freeing up mem in RAM, but then it's being written to the pagefile as it fragments. Since I am not a programmer - (I'm a hardware guy) it was my poor attempt to make sense of what's happening and to help.

os2wiz

on May 02, 2006

So the the bottom line is that quick solutions are very often NOT the BEST solutions. So avoid quick solutions which cause your customers unnecessary grief. Secondly testing without lesss than 2 GB pysical memory was a very dumb move. Inexcuseable. I have 1GB memoery and most people have 500 MB. Since I saw posts on how GC II
1.0 was optimized for lesser memory confiurations the same should have applied to your testing process for 1.1.
Assumption is the Mother of all f-ups as the vulgar expression goes and Stardock blew it royally here. Hopefully this will be an OBJECT lesson that Windows blinded developers will learn for a long time to come.

Dano13

on May 02, 2006

Actually I have always been scared of using defrag. On a large drive it is an overnight job, and if the power goes out your system is pretty much screwed ... no? Please correct me if I am wrong here ...

Dano

Welcome Guest! Please take the time to register with us.
There are many great features available to you once you register, including:

Richer content, access to many features that are disabled for guests like commenting on the forums.
Access to a great community, with a massive database of many, many areas of interest.
Access to contests & subscription offers like exclusive emails.
It's simple, and FREE!