Memory fragments

Published on May 1, 2006 By CariElf In GalCiv Journals

Our top priority since 1.1 came out has been resolving the memory and performance issues that people have been having. BoundsChecker didn't come up with any significant memory leaks, so that left us with checking the change logs to see what we might have done to cause the issues. We were obviously doing something wacky, because people with 4 GB page files were still getting out of virtual memory errors.

One of the changes we made was to the way that the shipcfg files were parsed. The shipcfg files are really just ini files, which I think Joe picked for the shipcfg format because ini files are supposed to be fast for reading and writing. They have two main disadvantages. The first is that you can't have an ini file greater than 64 kb on Windows ME and Windows 98; it won't be able to read anything past the 64 kb mark. The second is that you have to know what is the size of the longest section name, and the longest key name, or else know what all the section and key names are before you go to read in the data from the file. If you don't know the section and keynames, you have to call APIs to get all the section and keynames that are in the file, and you need a buffer to store them.

There are two ways to allocate memory: static and dynamic. Static means that you always allocate the same amount of memory, no matter how much of it you actually need to use. If you allocate too much, you'll be taking up memory you aren't using, but if you don't allocate enough, you'll run into problems because you don't have enough to use. Dynamic memory is allocated as you need it, when you need it. Since you're in charge of dynamic memory, you have to remember to deallocate (release) it. When you forget to deallocate it, it becomes a memory leak. As far as your program and OS are concerned, that memory is still being used and is unavailable to be allocated to something else. Another bad thing about dynamic allocation is that you can fragment memory.

I'm assuming that most of you have seen Windows' disk defrag program. If you haven't, you might want to go to Start->All Programs->Accessories->System Tools and run it, because your had drive probably needs it.

It will also give you a visual indication of what I'm talking about in this paragraph. When you create or copy files to your hard drive, the files are copied into consecutive blocks of memory. If you delete one or more of those files, that leaves a hole in memory. Depending on how big the hole is, it might get filled with other files. But if it's too small, it's just wasted. Disk defrag goes through your hard drive and tries to move stuff around so that it is stored more efficiently, with bigger blocks of available memory. Fragmentation can also happen in system memory, aka RAM. So even if you deallocate dynamic memory, there's a chance that the memory you released won't be used again by your program, so it keeps using more memory. When the program runs out of available RAM, it will start using virtual memory. Virtual memory is really just a section of the hard drive that is set aside for the operating system's use when it runs out of RAM.

So how does all of this relate to GC2's issues? The shipcfg files originally used static memory allocation for the buffer that was used to get the section and keynames, but if you added enough jewelry and components to the ship, the buffer wasn't big enough. We needed a quick way to fix this that wouldn't involve re-engineering the shipcfg code and that wouldn't make reading in the shipcfg files take longer. The quickest change to implement was to switch from using static allocation to using dynamic allocation. In order to make sure that the buffer was big enough, I suggested that we dynamically allocate a buffer that was the same size as the file. The bad thing about this solution was that it didn't take into account that we weren't saving the parsed data values in memory after reading in the file for the first time. Every time you built a colony ship, the colony ship cfg file was read in and a buffer was allocated and deallocated. So I'm thinking that we were probably fragmenting memory.

Last week, I wrote code to make GC2 saved the parsed values from the shipcfg files in memory, so that they would only have to be read in once. It would mean that we're hanging on to a little more memory than we were before, but it should cut down on the fragmentation. It definitely cuts down on the amount of time spent creating ship graphics, which you will notice when loading a save game. If it's not enough, we may still have to change the shipcfg file format, but keeping the parsed results in memory will help keep load times down. I wrote the new shipcfg code in such a way that I should only have to replace one function if we need to switch the file format, the one that actually reads in the data. The code that uses the stored data to put together a ship with all its components will not need to be changed.

Another problem area is the save game code. Writing to the hard disk is slow, so it's quicker to create the file in memory and then write it out to the file in one fell swoop rather then writing out each datum as you go. Since the exact file size of a given save game is unknown, the save game code uses dynamic memory allocation. Each object in the game (ie ships, planets, civs, etc) has its own function to create a block of memory containing all the data it needs to store, which it then passes to the main save game function, and is added to the main block. This is using the same code as in GC1, but we had less dynamic data in GC1. Originally, all of the buffers started out as 1 kb and whenever the buffers needed to increase in size, they would allocate their current size + 1 kb and copy the data from the old buffer to the new buffer, then deallocate the old buffer. The process of growing and copying the buffers was taking up more time than the actual saving of data, and I needed a way to improve performance without doing major surgery to the save game code before version 1.0 came out. So I did some profiling on how big the buffers were for a gigantic galaxy in the first few turns of the game and how much they grew, and used those numbers to change the initial buffer sizes and how much they grew by for each data object. This was, admittedly, more of a band-aid than an actual fix.

Apart from adding some new things that needed to be saved, I don't think that we've really touched the save game code much. However, it is still fairly inefficient because of all the buffer growing and copying. So the next change I have started to make is to make all the data objects use one buffer. Once that is done, I can make further optimizations to the code like initializing the buffer size based on the galaxy size, and see if it does more good than harm to keep the buffer in memory so that it doesn't have to allocate and deallocate 2-13 MB (or more) every time the game saves. At the very least, making everything use one buffer should cut down on fragmentation and make the saving go quicker. I will also be reviewing the code to make sure that only necessary data is being saved rather than being recalculated, in an effort to cut down loading time.

Once I've finished making our memory usage more efficient, I'll start working on the modding stuff again.

Edit: Ok, since I'm getting e-mails and comments about this, I would like to clarify that I am not blaming your hard drives for causing the crashes. The point of this article is that I am working on resolving the memory issues. The comment about running disk defrag was meant as a general statement that you should regularly defrag your hard drives, and to provide a visual representation of what is happening when memory is fragmented.

Update: In my sticky thread here Link I put instructions and a link for an unofficial test exe.
--Cari

Comments (Page 1)

Lord Reliant

on May 01, 2006

Let me be the first to give you a kudos for your hard work. Just last night as I was playing GalCiv2 I realized one of the few areas that seemed "slow" on my computer was loading/saving. Tweaking this and cutting down on memory fragmentation should tighten this are up even further. I am continually impressed with Stardock's attention to details in ALL areas of their game. It's also great to see that Brad isn't the only guy doing high and low level stuff.

So, I'm assuming these changes will be coming for 1.2, then, right? Or will there be a 1.1.x where it's simply a bug/tweak patch?

spleeze_

on May 01, 2006

Ack, memory fragmentation. I’m surprised that you didn’t just stick with xml for shipconfig, and use the xml parser you used for the rest of the game files. Did your xml parser really gobble up so much more processor time then the ini parser?

On the topic of modding (not really, but…), I have noticed that previously (as in, during initial patches), the ship builder would show the appropriate icon for a new ship hull that I’ve just imported. I could go into the ship builder, click new ship, then “large” hulls, and the icons listed would all be miniatures of my imported hulls. This seems no longer to be consistent, in that some hulls will show a proper “hull” thumbnail, while some show the old “original” hull thumbnails. Note that once the ship has been built and saved, the “ship” thumbnails do appear to display properly in the ship builder selection screen.

Let me show you an example. The first of the large hulls is stock, while the second and third are my custom imports.

link

Here I have selected the third Large ship hull, which does have a proper thumbnail in both the browser on the left, and the “hull contents” list at the top.

link

However here I have selected the second Large ship hull, but both thumbnails are of the old mesh that has been since overwritten. These must be cached somewhere? This problem persists across game restart. I am running 1.1.

Also, a suggestion for modding capability: Allow modders to be able to change the icons of ships / fleets when in tactical view. I would like my star trek ships to show up with a star trek logo when they are zoomed out, like you see in the ST movies. This would obviously have to be only for the player’s race, and not for every race in the game (as currently, they all share the same thumbnail, just a different color). Nothing critical, but it would be nice ; )

-Brian

Soundbyte

on May 01, 2006

I did indeed find my hard drive heavily fragmented, but to me, the save/load times are normal. It's the ending program that causes my computer to freeze up, sometimes to the point I have to control-alt-del to end the process prematurely. I do notice the problems especially when A: fleets of large ships are on the screen (more than 2 fleets) and B: The ai has a large number of ships, and spends a lot of time moving all of them.

Hope this helps.

(EDIT: boogie, you wouldnt be hunted down... You'd quite literally "die in a fire." Just clarifying

)

Erdrick84

on May 01, 2006

I love you guys!

have my babies?

ScottTykoski

on May 01, 2006

grrr Went to update the post and accidentally erased it. I forgot that Brad would be out for some time, so we're going to work on getting this stuff fixed and update as soon as possible.

Soundbyte: hehe Yeah, but if we're burned at the stake, then who's going to do the bug fixes

Lord Reliant

on May 01, 2006

That's great there will be a stability/fix point release coming out. I did have one strange happening as well. I had been playing GalCiv2 for about 2 hours. I'm not sure what triggered it, but suddenly the game just quit. No error, no warning. Just quit. I'm assuming some of these memory-related problems would generate an error, or would they manifest themselves in the manner I described?

Also, are you seeing more memory-related issues now that ships are being better built and people are customizing them more with the enhancements to the ship design interface in 1.1?

Finally, in response to SoundByte above, I never have problems quitting. In fact, quitting is the fastest thing about the whole game. As soon as I choose to quit I'm dropped to the desktop in about 1 second. That's odd it's almost an opposite scenario than what I described.

CariElf

on May 01, 2006

Erdrick84,

Only if you pass our rigorous DNA testing.

CariElf

on May 01, 2006

spleeze_,

I'm not sure if Joe did any testing to see if the ini files were that much faster than the xml parser, but we may find out if my changes aren't enough to stop the memory fragmenting.

Regarding the thumbnails, I'm fairly certain that we don't save the thumbnails to disk anymore, so maybe it's not using the correct model name. I'll have to check into that.

Soundbyte,

The suggestion to defrag the hard disk was more a general suggestion; it's generally a good idea to defrag your hard disk once a month. It also gives less technical people a visual representation of fragmented memory.

Lord Reliant,

I suppose it's possible. The next time it happens, check your debug.err file to see if it thinks that the Quit button was hit. We've had a bug that has been around since the early betas where the game thinks that the quit button on the title menu was clicked, so it closes down the game. The problem is, it's pretty much impossible to reproduce so we haven't figured out how to fix it yet. It's pretty rare, though, and I think that it's more rare since I filtered out the windows messages that we weren't using.

Vandenburg

on May 01, 2006

Only if you pass our rigorous DNA testing.

Because of the profound knowledge of genetic manipulation I've got from researching Xeno Farm Construction III this shouldn't be a problem of him.

Btw. nice to know that you people work hard to solve this problem.

Lord Reliant

on May 01, 2006

CariElf-

If that happens again, I'll check debug.err. I didn't know this existed (I hadn't looked yet), so I imagine if anything else odd happens I'll check there as well. I really suspect that the bug you described is what happened to me, since it acted as though I had quit the game. The only thing I could think of is if it was somewhat time-based as I had been playing for some time, but other than that I can't think of a reason that would have triggered that. Unless Brad's AI is now using that as a counter-measure to clever human moves that weren't saved yet

Suralle

on May 01, 2006

Nothing big, just Thank You.

W/R
Suralle Straykat
Kat Lord @ Large

Soundbyte

on May 01, 2006

Boogiebac: Easy. We'll just do it one by one, in full sight of you guys while at work in your cubicles... excellent motivation, right? (Yes, I play evil choices every time, all the time)

CariElf: Actually, I do defrag once a month at least. But since the new patch, and boosting my pf, over this past week my drive somehow became HEAVILY fragmented. Concidence?

Erdrick84

on May 01, 2006

I sent some DNA in the mail.

I'll want my pinkie toe back when youre done with it.

whismerhill

on May 01, 2006

Once I've finished making our memory usage more efficient, I'll start working on the modding stuff again.

sorry to bring this up but when will known easily repeatable "problems" be fixed as opposed to modding stuff ?

I mean : modding is great but shouldn't fixing "problems" before be a top priority ? (even if they are considered small ..)
(like the one where the first turn after loading a game is not correct)

that said improving memory usage is great !

(to rephrase, I'd like to know where these are on your schedule)
thanks.

CariElf

on May 01, 2006

whismerhill,

Someone somewhere is always going to have a bug with some kind of pc software. We do what we can to stamp out the big ones and as many little ones as possible while also adding features to keep people interested in the game. Modding, in particular, kept people interested in GalCiv1 long past its release date.

Welcome Guest! Please take the time to register with us.
There are many great features available to you once you register, including:

Richer content, access to many features that are disabled for guests like commenting on the forums.
Access to a great community, with a massive database of many, many areas of interest.
Access to contests & subscription offers like exclusive emails.
It's simple, and FREE!