Memory fragments

Published on May 1, 2006 By CariElf In GalCiv Journals

Our top priority since 1.1 came out has been resolving the memory and performance issues that people have been having. BoundsChecker didn't come up with any significant memory leaks, so that left us with checking the change logs to see what we might have done to cause the issues. We were obviously doing something wacky, because people with 4 GB page files were still getting out of virtual memory errors.

One of the changes we made was to the way that the shipcfg files were parsed. The shipcfg files are really just ini files, which I think Joe picked for the shipcfg format because ini files are supposed to be fast for reading and writing. They have two main disadvantages. The first is that you can't have an ini file greater than 64 kb on Windows ME and Windows 98; it won't be able to read anything past the 64 kb mark. The second is that you have to know what is the size of the longest section name, and the longest key name, or else know what all the section and key names are before you go to read in the data from the file. If you don't know the section and keynames, you have to call APIs to get all the section and keynames that are in the file, and you need a buffer to store them.

There are two ways to allocate memory: static and dynamic. Static means that you always allocate the same amount of memory, no matter how much of it you actually need to use. If you allocate too much, you'll be taking up memory you aren't using, but if you don't allocate enough, you'll run into problems because you don't have enough to use. Dynamic memory is allocated as you need it, when you need it. Since you're in charge of dynamic memory, you have to remember to deallocate (release) it. When you forget to deallocate it, it becomes a memory leak. As far as your program and OS are concerned, that memory is still being used and is unavailable to be allocated to something else. Another bad thing about dynamic allocation is that you can fragment memory.

I'm assuming that most of you have seen Windows' disk defrag program. If you haven't, you might want to go to Start->All Programs->Accessories->System Tools and run it, because your had drive probably needs it.

It will also give you a visual indication of what I'm talking about in this paragraph. When you create or copy files to your hard drive, the files are copied into consecutive blocks of memory. If you delete one or more of those files, that leaves a hole in memory. Depending on how big the hole is, it might get filled with other files. But if it's too small, it's just wasted. Disk defrag goes through your hard drive and tries to move stuff around so that it is stored more efficiently, with bigger blocks of available memory. Fragmentation can also happen in system memory, aka RAM. So even if you deallocate dynamic memory, there's a chance that the memory you released won't be used again by your program, so it keeps using more memory. When the program runs out of available RAM, it will start using virtual memory. Virtual memory is really just a section of the hard drive that is set aside for the operating system's use when it runs out of RAM.

So how does all of this relate to GC2's issues? The shipcfg files originally used static memory allocation for the buffer that was used to get the section and keynames, but if you added enough jewelry and components to the ship, the buffer wasn't big enough. We needed a quick way to fix this that wouldn't involve re-engineering the shipcfg code and that wouldn't make reading in the shipcfg files take longer. The quickest change to implement was to switch from using static allocation to using dynamic allocation. In order to make sure that the buffer was big enough, I suggested that we dynamically allocate a buffer that was the same size as the file. The bad thing about this solution was that it didn't take into account that we weren't saving the parsed data values in memory after reading in the file for the first time. Every time you built a colony ship, the colony ship cfg file was read in and a buffer was allocated and deallocated. So I'm thinking that we were probably fragmenting memory.

Last week, I wrote code to make GC2 saved the parsed values from the shipcfg files in memory, so that they would only have to be read in once. It would mean that we're hanging on to a little more memory than we were before, but it should cut down on the fragmentation. It definitely cuts down on the amount of time spent creating ship graphics, which you will notice when loading a save game. If it's not enough, we may still have to change the shipcfg file format, but keeping the parsed results in memory will help keep load times down. I wrote the new shipcfg code in such a way that I should only have to replace one function if we need to switch the file format, the one that actually reads in the data. The code that uses the stored data to put together a ship with all its components will not need to be changed.

Another problem area is the save game code. Writing to the hard disk is slow, so it's quicker to create the file in memory and then write it out to the file in one fell swoop rather then writing out each datum as you go. Since the exact file size of a given save game is unknown, the save game code uses dynamic memory allocation. Each object in the game (ie ships, planets, civs, etc) has its own function to create a block of memory containing all the data it needs to store, which it then passes to the main save game function, and is added to the main block. This is using the same code as in GC1, but we had less dynamic data in GC1. Originally, all of the buffers started out as 1 kb and whenever the buffers needed to increase in size, they would allocate their current size + 1 kb and copy the data from the old buffer to the new buffer, then deallocate the old buffer. The process of growing and copying the buffers was taking up more time than the actual saving of data, and I needed a way to improve performance without doing major surgery to the save game code before version 1.0 came out. So I did some profiling on how big the buffers were for a gigantic galaxy in the first few turns of the game and how much they grew, and used those numbers to change the initial buffer sizes and how much they grew by for each data object. This was, admittedly, more of a band-aid than an actual fix.

Apart from adding some new things that needed to be saved, I don't think that we've really touched the save game code much. However, it is still fairly inefficient because of all the buffer growing and copying. So the next change I have started to make is to make all the data objects use one buffer. Once that is done, I can make further optimizations to the code like initializing the buffer size based on the galaxy size, and see if it does more good than harm to keep the buffer in memory so that it doesn't have to allocate and deallocate 2-13 MB (or more) every time the game saves. At the very least, making everything use one buffer should cut down on fragmentation and make the saving go quicker. I will also be reviewing the code to make sure that only necessary data is being saved rather than being recalculated, in an effort to cut down loading time.

Once I've finished making our memory usage more efficient, I'll start working on the modding stuff again.

Edit: Ok, since I'm getting e-mails and comments about this, I would like to clarify that I am not blaming your hard drives for causing the crashes. The point of this article is that I am working on resolving the memory issues. The comment about running disk defrag was meant as a general statement that you should regularly defrag your hard drives, and to provide a visual representation of what is happening when memory is fragmented.

Update: In my sticky thread here Link I put instructions and a link for an unofficial test exe.
--Cari

Comments (Page 5)

os2wiz

on May 04, 2006

I never used the word dumb or stupid. So forget the hyperbole.

os2wiz

on May 04, 2006

I was not criticizing the effort for the 1.0 release. I specifically referred to the 1.1 release. Please get your facts straight.

os2wiz

on May 04, 2006

I made a comment on the first issue you raised. It is a completely legitimate criticism and I was flamed for it. I have no gods or heros that I worship. There is nothing in this universe that is perfect. All should be able to be scrutinized and critiiczed, including me. I believe the antagonism exists because I said "there is no excuse" for Stardock to have tested 1.1 development only on 2gigabytes of memory. I will not temper that remark, it is entirely fair. I also said that assumptions are the mother of all f-- ups. That line came from a Steven Seagal movie. I said it
to reinforce the importance of what I had said previously NOT to insult and not to belittle. I wanted to make sure the
importance of correcting the methodology of development testing, so this could be avoided in the future. Assuming
responsibility is an important step in correcting errors. I had not seen that being done here ina frank way. If I had I would never had said anything. I was not attempting to poor salt on a wound . I hope this clarifies my remarks. Now PLEASE it is time to move on.

CariElf

on May 04, 2006

All right, that's enough.

There will be no more discussion from either side about os2wiz's replies to this journal. There are already too many off topic replies in this thread.

codingmonkey

on May 04, 2006

Hehe Vulcannis you beat me to it

Anyways you can find more info about Prolog here: Link

I can't remember from my Comparative Programming Languages classes because we never actually used Prolog or Lisp. We spent most of our time on functional programming, which drove me up the wall.

Yeah I had to take a similar course to what you took Cari, or at least I think it is similar. We learnt about three different languages: Scheme, ML and Prolog. Both Scheme and ML are functional programming languages and Prolog, as Vulcannis said, is a logic programming language. Also, my main problems with those three languages is just the complete lack of a good debugger (at least I couldn't find one). So if you haven't used them, your lucky. Also, I couldn't stand all those parenthesis

(although if you ever write Scheme/LISP code, Kate (Linux application) is awesome, it colours the parenthesis for you so it's easy to match them up)

Prolog is very suitable for searches, and I'm not really sure what kind of AI algorithms Stardock uses (whatever they are, their _really_ good

Half the business devs I work with would barely grok what it is, and the language people tend to view it as just another tool in the box, albiet probably a more fundamental one than say iteration.

I don't really hate recursion too much because there are a lot of situations when it is very useful, but in languages like Prolog where an iterative solution makes sense and you have to use recursion.. well I'll refrain from cursing.

Anyways, always fun talking programming concepts and confusing people on the forums

codingmonkey

on May 04, 2006

There are already too many off topic replies in this thread.

Hmm sorry for the last post then

CommanderGizmo

on May 04, 2006

I have written my fair share of small apps. Being small, they rarely have any issues with memory fragmentation, though memory usage and leakage does (of course) come up. In my humble thought process, it seems to me that if memory fragmentation was at fault, my system memory usage would not be reporting near the limit when the crash occurs, but well under it. This would be because the problem is lack of contiguous blocks of memory within the lower 2GB hard limit area, not due to it actually being in use. I am thinking the crash is due to trying to access a memory block above the 2GB wow64 limit. Is this correct, or am I off in left field? My system is defiantly showing right at 2GB of RAM being taken (not using much virtual at this point), and only since the 1.1 update. It crases when the save feature creates that last little bit to push it over the 2GB mark.

Anyone set me on the right track?

CariElf

on May 04, 2006

SuperGeek,

If I recall correctly, memory fragmentation can cause 'out of memory issues' because if it can't allocate memory from the areas of memory that have been deallocated, so it goes to allocate memory from another section. So you could end up not being able to allocate the block of memory because there is not a contiguous section of memory left that is big enough to allocate it.

All of the debug.err and crash logs that I received indicated that it was crashing after it failed to allocate memory, because malloc returned a NULL pointer, and our code was not happy with that (which is something I'm trying to figure out). However, on the test machine that we setup yesterday with only 256 MB of RAM and a 766 MB page file, we managed to make it go over the maximum virtual memory limit before saving, so it just died. No SE log or even a Windows error.

Whether or not we are actually fragmenting memory, the changes that I made to the save game help with the memory issue because there is not as much allocating, copying, and deallocating going on. Since it has to keep the old buffer in memory while it is creating the new one before it can copy the memory, that's slightly more than double the size of the old buffer hanging around in memory. On the lower end machines, that's a pretty painful situation.

CommanderGizmo

on May 04, 2006

Hmm... That is where I get confused. If it is simply contiguous memory blocks that are unavailable, wouldn't the game be showing less than the maximum memory at time of error? I guess if the largest block of memory you are trying to lock is pretty small then it wouldn't cause an error until it is very close to the limit, thus producing what I am seeing. Do you know how large that would be during a save?

CommanderGizmo

on May 04, 2006

Just ran a test with the latest test exe. Hope this information is helpful.

I opened the same save game file (restored file from other location before each load) with latest official release and text exe. I then ran the game (gigantic map, all races, mid game) for 5 turns. I have duel screen setup with the game running windowed. I have task manager running on second screen at the same time.

After loading, both games took within 20MB of each other in memory. I noticed (as stated before) that memory only seems to grow when the end of a turn is running, though I havn't checked when building a new ship and such. The official exe made the memory grow about 20 - 30 MB per turn. The latest test exe grew memory by 15 - 20 MB per turn, with slightly greater speed through the end turn calculations. That little bit helps alot, but will still overrun the 2GB limit quickly. It is definatly getting better. Thanks for the hard work!

Ezlbeth

on May 05, 2006

Oh disregard me. Wanted to point out the 2gb limit for windows being the issue not low end machines.

If you guys want it takes me about 15-20 minutes to load/save a gigantic galaxy the first time (secondary saves are faster).

Although that might just be perceptions, I have really large load times if that might help with something, let me know what to look for.

Alfb

on May 05, 2006

Cari, out of curiosity, what platform are you guys developing Gal Civ2 on? I assume C++ and that probably means Visual C++. What version? The main reason I ask is because of the points Vulcannis brings up. It also brings up some memories of improvements in Garbage Collection that have been made in later versions.

os2wiz:

You just don't get it. I agree that Stardock should have thought to test at the recommended config but that does not justify the way you state it. Your later attempts to back up a little bit just make it worse. OS2 is not the issue here and had way too many of its own problems that you seem to have forgotten. I could not even get it to install in many cases. But I digress, you need to learn there is a professional way to make your point if you want to be respected. Surely you do want respect and if not, you are not worth the time I am taking to respond. I have been a sometimes vocal critic of Stardock but I have tried to make it be cases that are justified and in ways that do not insult. At least remember that you will have a better chance of getting what you want here by working with them rather than the hellfire brimstone attitude you currently spout. The sooner you figure that out, the sooner you will get more of what you and I want, a better game.

player1_fanatic

on May 06, 2006

Alt-Tabing crashes my game in 50% of cases.

Now crashes do happen in some other cases, but not often.

But, Alt-Tabing is something that definetly needs to get looked up.

Peace Phoenix

on May 06, 2006

Alt-Tabing crashes my game in 50% of cases.

In fullscreen or in windowed mode? BTW, if you set a windowed mode corresponding to your destop resolution, you will have something like fullscreen, but better at handling ALT+TAB

player1_fanatic

on May 06, 2006

Fullscreen, same resolution as my desktop.

Welcome Guest! Please take the time to register with us.
There are many great features available to you once you register, including:

Richer content, access to many features that are disabled for guests like commenting on the forums.
Access to a great community, with a massive database of many, many areas of interest.
Access to contests & subscription offers like exclusive emails.
It's simple, and FREE!