Memory fragments

Published on May 1, 2006 By CariElf In GalCiv Journals

Our top priority since 1.1 came out has been resolving the memory and performance issues that people have been having. BoundsChecker didn't come up with any significant memory leaks, so that left us with checking the change logs to see what we might have done to cause the issues. We were obviously doing something wacky, because people with 4 GB page files were still getting out of virtual memory errors.

One of the changes we made was to the way that the shipcfg files were parsed. The shipcfg files are really just ini files, which I think Joe picked for the shipcfg format because ini files are supposed to be fast for reading and writing. They have two main disadvantages. The first is that you can't have an ini file greater than 64 kb on Windows ME and Windows 98; it won't be able to read anything past the 64 kb mark. The second is that you have to know what is the size of the longest section name, and the longest key name, or else know what all the section and key names are before you go to read in the data from the file. If you don't know the section and keynames, you have to call APIs to get all the section and keynames that are in the file, and you need a buffer to store them.

There are two ways to allocate memory: static and dynamic. Static means that you always allocate the same amount of memory, no matter how much of it you actually need to use. If you allocate too much, you'll be taking up memory you aren't using, but if you don't allocate enough, you'll run into problems because you don't have enough to use. Dynamic memory is allocated as you need it, when you need it. Since you're in charge of dynamic memory, you have to remember to deallocate (release) it. When you forget to deallocate it, it becomes a memory leak. As far as your program and OS are concerned, that memory is still being used and is unavailable to be allocated to something else. Another bad thing about dynamic allocation is that you can fragment memory.

I'm assuming that most of you have seen Windows' disk defrag program. If you haven't, you might want to go to Start->All Programs->Accessories->System Tools and run it, because your had drive probably needs it.

It will also give you a visual indication of what I'm talking about in this paragraph. When you create or copy files to your hard drive, the files are copied into consecutive blocks of memory. If you delete one or more of those files, that leaves a hole in memory. Depending on how big the hole is, it might get filled with other files. But if it's too small, it's just wasted. Disk defrag goes through your hard drive and tries to move stuff around so that it is stored more efficiently, with bigger blocks of available memory. Fragmentation can also happen in system memory, aka RAM. So even if you deallocate dynamic memory, there's a chance that the memory you released won't be used again by your program, so it keeps using more memory. When the program runs out of available RAM, it will start using virtual memory. Virtual memory is really just a section of the hard drive that is set aside for the operating system's use when it runs out of RAM.

So how does all of this relate to GC2's issues? The shipcfg files originally used static memory allocation for the buffer that was used to get the section and keynames, but if you added enough jewelry and components to the ship, the buffer wasn't big enough. We needed a quick way to fix this that wouldn't involve re-engineering the shipcfg code and that wouldn't make reading in the shipcfg files take longer. The quickest change to implement was to switch from using static allocation to using dynamic allocation. In order to make sure that the buffer was big enough, I suggested that we dynamically allocate a buffer that was the same size as the file. The bad thing about this solution was that it didn't take into account that we weren't saving the parsed data values in memory after reading in the file for the first time. Every time you built a colony ship, the colony ship cfg file was read in and a buffer was allocated and deallocated. So I'm thinking that we were probably fragmenting memory.

Last week, I wrote code to make GC2 saved the parsed values from the shipcfg files in memory, so that they would only have to be read in once. It would mean that we're hanging on to a little more memory than we were before, but it should cut down on the fragmentation. It definitely cuts down on the amount of time spent creating ship graphics, which you will notice when loading a save game. If it's not enough, we may still have to change the shipcfg file format, but keeping the parsed results in memory will help keep load times down. I wrote the new shipcfg code in such a way that I should only have to replace one function if we need to switch the file format, the one that actually reads in the data. The code that uses the stored data to put together a ship with all its components will not need to be changed.

Another problem area is the save game code. Writing to the hard disk is slow, so it's quicker to create the file in memory and then write it out to the file in one fell swoop rather then writing out each datum as you go. Since the exact file size of a given save game is unknown, the save game code uses dynamic memory allocation. Each object in the game (ie ships, planets, civs, etc) has its own function to create a block of memory containing all the data it needs to store, which it then passes to the main save game function, and is added to the main block. This is using the same code as in GC1, but we had less dynamic data in GC1. Originally, all of the buffers started out as 1 kb and whenever the buffers needed to increase in size, they would allocate their current size + 1 kb and copy the data from the old buffer to the new buffer, then deallocate the old buffer. The process of growing and copying the buffers was taking up more time than the actual saving of data, and I needed a way to improve performance without doing major surgery to the save game code before version 1.0 came out. So I did some profiling on how big the buffers were for a gigantic galaxy in the first few turns of the game and how much they grew, and used those numbers to change the initial buffer sizes and how much they grew by for each data object. This was, admittedly, more of a band-aid than an actual fix.

Apart from adding some new things that needed to be saved, I don't think that we've really touched the save game code much. However, it is still fairly inefficient because of all the buffer growing and copying. So the next change I have started to make is to make all the data objects use one buffer. Once that is done, I can make further optimizations to the code like initializing the buffer size based on the galaxy size, and see if it does more good than harm to keep the buffer in memory so that it doesn't have to allocate and deallocate 2-13 MB (or more) every time the game saves. At the very least, making everything use one buffer should cut down on fragmentation and make the saving go quicker. I will also be reviewing the code to make sure that only necessary data is being saved rather than being recalculated, in an effort to cut down loading time.

Once I've finished making our memory usage more efficient, I'll start working on the modding stuff again.

Edit: Ok, since I'm getting e-mails and comments about this, I would like to clarify that I am not blaming your hard drives for causing the crashes. The point of this article is that I am working on resolving the memory issues. The comment about running disk defrag was meant as a general statement that you should regularly defrag your hard drives, and to provide a visual representation of what is happening when memory is fragmented.

Update: In my sticky thread here Link I put instructions and a link for an unofficial test exe.
--Cari

Comments (Page 4)

Drengin

on May 03, 2006

I definately concur that the extra ship components were the severe memory problem. It was very obvious in playing a game with and without those very detailed ship designs. WIthout them the game flew on my machine, with them it started to crawl during the save games and the load times took minutes instead of seconds.

This i concur. As an avid ship designer for the game i agree that too much jewelry on a ship increases the load times for the ship. With the latest batch of ships i created (the USM pack #1) the ships appear in the game but there is a delay in showing them in the ship builder window and in the update screen, this is esp obvious in the constructor that has an additional 30 components on the exterior of the ship as well as mounts for those, the two ships on the top as well as the ship itself. In total there must be close to 100 jewelry parts on the ship alone.

That said i am still getting a huge delay in loading the game even with the ships removed. I was able to reduce this somewhat by setting the mods check box to 'no' so the game is no longer checking the mods directory as well.

As an avid designer, as i have mentioned before, i would like to propose a 'cap' on the amount of jewelry you can use on any one ship. This needs to be no higher than 50 pieces for any one ship as this is sufficient to make a good ship. More than this causes most peoples games to slow to a crawl when you fleet the ships or have more than a dozen or so on the screen at once in normal view.

My $0.02

DG

Wahngrok

on May 03, 2006

I will not apologize for my comments which were accurate, insiteful, and wryly humorous. I am not in the habit of kissing butt to have my views respected. They stand on their own merit.

I don't agree to your first but on the other statements. Ah, the irony.

wushudragon69

on May 03, 2006

Carielf has a very thin skin, indeed, if my caustic comments, which were entirely accurate, get under her skin.

Maybe you are mistaking the fact that you have a thick skull and she does not? Anyways, anyone who still clings to a dead and buried OS like os/2 really has no point in arguing business matters. And Yes, I have used it before, actually at my place of business we had to use it for several small tasks on some 386 machines up until about 5 years ago? These days the only logical choices is between windows and *nix based OS's. An OS with no vendor support and absolutely no user base has no place anywhere except some die-hard who is still clinging to antiquated software in defiance to the world that has moved on to better solutions. In any case that does not sound like someone who should be giving advice to a person in charge of developing a hugely successful software title.

os2wiz

on May 03, 2006

Your arguement is specious. I never said I primarily or exclusively use os/2 which, under its current flavor, is called eComStation v 1.2 . I primarily use windows XP. I rarely use os/2. That does not alter the fact that its memory management is far superior to anything that has ever come out of Redmond, Washington. And that matter has little relevance to my remarks in the flawed testing conducted for Galciv II v 1.1 . I was not nor am I now attempting to mock
or belittle any individual at Stardock. I think it is one of the better sotware companies and has very good personnel
including Carielf. That does not mean they didn't blunder somewhat in this project . I respect their intelligience and ability, but nevertheless they did blunder. Hopefully some introspection regarding this will help improve their future development. That is how all of us as individuals and as vibrant enterprises improve: by learning from our errors.
After all I would never have achieved my exultant status in life had I not learned from my errors as a mortal. That iis how I have achieved immortality. That is what my victories in Galciv have said in anycase at the end of the games.
Please do not take this seriously. I am actually NOT Beyond Immortality as the game suggests.
Food for thought.

CommanderGizmo

on May 03, 2006

This thread is here for the memory issues at hand, not your short sighted egotistical promotion of your own ideas. Please shut up or focus your attention on the polite and hopefully helpful conversation that existed here before your presence. I'm sure everyone will appreciate it.

Supreme Shogun

on May 03, 2006

OS2WIZ: The "uproar" is over the tone of your post. Lets see, you called Stardock dumb, accused them of making the mother of all F---ups and also stated they screwed up royally. That's insulting, plain and simple. You show no respect or decency. You could have stated facts without those insults, AND you didn't. Not to mention, You are wrong in YOUR ASSUMPTIONS.

I remember right before initial release a statement by Brad that they went to great lengths testing on variety of machines different vid cards and memory, specifically soliciting info/performance results from anyone still having a 32 meg card --lowest supported card. Stardock went to lengths to support the LOW end as well as the high and it was tested before release.

os2wiz

on May 03, 2006

I was not egotistically promoting anything. And you certainly will not squelch me. The thread was about the 1.1 memory problem. I have the problem big-time. If you did not attempt to muzzle me would have already moved on by now. So stop your personal attacks. I never made one in any of my posts. I posted a factual description of my own
experiences with the memory crashes on the other similar thread in the forum area. I was not seeking a debate about os/2 or windows, simply making an observation about the poor memory management. it required no emotional outburst from you at all.

CommanderGizmo

on May 03, 2006

Please see [link url="https://forums.galciv2.com/?ForumID=162&AID=115094#891298"]this">Link thread (which you've posted in) for an example of how to present the idea you have had properly. The post is written by Xentax and states as follows:

Couple comments - I mean these in a constructive way, so please don't take them as a flame.

1) I am all for giving devs fast, beefy hardware on which to develop the game. You literally can't put a price tag on maximizing developer productivity by making sure they have a fast box, 2 or even 3 displays, and whatever their personal idea of an ideal working environment happens to be (quiet, or music, dim light, bright light - whatever it is).

HOWEVER, your *testers* should have more than just one of those machines; they should also have a range of hardware configs to test on - and one or more of those should exactly match the game's posted minimum specs. There's no excuse to not know *exactly* how your product runs on the minimum and recommended configurations (as well as "maxed out" hardware). I realize Stardock may not have the excess budget for a hardware lab with a hundred different configurations, but you definitely need to get a few machines with low-end specs (on the plus side, you ought to be able to get several of these for the prices of a single dev box ). A natural corrolary is to be sure to test several of the most common video card chipsets (even if you're partnered with NVidia or ATI), at a variety of performance points.

2) Memory fragmentation: At the risk of getting way too technical, is there any particular reason you're not building the game with a non-default heap manager (e.g. SmartHeap)? It may take some work to plug in, depending on how you've done your existing memory management, but you shouldn't have to reinvent the wheel to address this. You may even be able to use the memory allocators from something like ACE if you don't want to shell out the $$ for a commerical solution like SmartHeap (I don't remember the exact licensing terms for the ACE libraries, but I'm pretty sure there's at least one commercial option).

-X

CariElf

on May 03, 2006

I have had very long save times since before the 1.1 beta. It never crashed though until recently. By the way its nottoo difficult to pull out 1GB of memory from the developers computer. It takes about 3 minutes to open the case pop out the the one or two memory modules involved and close the case again. I think my remarks were not only on the money, but also humorously sarcastic. You failed to note the capitalized OBJECT lesson and the "windows -blinded "developers. I also would enjoy pointing out. That os/2 uses memory FAR more efficiently than windows, and that the same code written to the os/2 platform would not have required the same physical memory and the swap file works far better than the so-called virtual memory of windows. My remarks were actually a triple entendre: a swipe at Stardock on the specific problen, a swipe using the name of Stardock products, and a swipe at the winows platform as still quite inept in certain api's and efficiency of resource utilization.

The problem with sarcasm in writing is that it can be difficult to distinguish if it IS actually sarcasm. The fact that you were swiping at Stardock's products names did not serve to distinguish your comments as sarcasm, and the fact this "humorous swipe" followed insulting language made me fail to see any humor in them at all. I don't expect you or anyone else to kiss up to me. As I said before, I don't mind criticism. I do mind when people give the criticism in an insulting and/or condencending tone. Your reply had all the makings of a flame, and it made me mad to the point that I flamed back, which is something that I hardly ever do.

Cari does the test build rectify the issue with the saved game files not loading? Or is that issue something else completely?

Yes.

Supreme Shogun

on May 03, 2006

A) You were condescending in your tone, "promoting" That you are right and others are wrong

I really, generally don't care what peoples opinions are BUT these were unwarranted attacks on Stardock and its employees.
C) You are beginning to miss the point and go off on a tangent, maybe you already have. You're right its about 1.1 version and the memory leak, its not about you. I say voice your opinion so long as you quit the outright insults/flames.
Dissension is possible but we don't have to swear, flame, and insult. Again, it's not about you.
D) Your warped sense of humour is harder than most to discern. I certainly didnt read any sarcasm into it, it was taken point blank and I'd say most readers also did the same. So if it looks like a lot of people are hostile to you, its because we/they didn't get your jokes and took your post in the literal.
E) I've not attacked you, but I did go back and see there was an OS2 comment

Everyone should stop being insulting . While you may not have insulted a single non-Stardock forum member, Yes you did personally attack Stardock and the developer team. I can't believe you are so BLIND not to see your own comments being inflammatory. Even so, "retaliation" or flaming you back is not cool either. Everyone should take a deep breath, calm down and "just let it go". Stardock and crew will tackle the problem.

To CariElf: Again thanks for the Dev Journal. No offense to Frogboy intended, but he hogs the spotlight just a wee bit.
99.8% of them are his ? The other developers, yourself included should be heard from more!
I love the insight and ability to see what other folks at Stardock is doing. Please, continue to do some from time to time. Also if its feasible, unchain the other developers from their cubicles (j/k) once in a while and have them do one.
I suspect Brad wants to do the majority of them and have said certain things and show certain things (talking points ), I'd say to that that's cool too but you can do both

codingmonkey

on May 03, 2006

I don't mean to jump into this little flamewar but I would like to point out some things to os2wiz. I'm not entirely sure if os2wiz is a developer or not but as a developer myself, I know exactly how Stardock employees feel. When you develop a piece of software you can test only so much. There will _always_ be bugs, _always_. It seems to me that Stardock has done a lot of testing and on multiple machines before they had released anything.

However, even if they did not, you have to keep in mind that it is very possible that Stardock did not have the budget to buy 100s of different machines to test on. In fact, taking a look at this wiki page, Link, it seems that Stardock has only 25 or so employees. That's a rather small company and probably reinforces my view. Again, they did have a long beta and it seems that people did not complain about this memory problem. Furthermore, os2wiz, you have to keep in mind that in the real world people have deadlines and so most of the time a product is pushed out with some risk of bugs still in the program.

Lastly, I would like to point out, as others already have, there are several ways in which one can make comments: 1. one can use a polite tone and give constructive criticism which will most likely be considered by the person/people that you are criticizing or 2. one can be rude about their comment at which point I think the person will just ignore anything you say. There are a lot of people who seem to think that #2 is OK, but in the real world this is not acceptable and will only make people dislike (as is clear from this forum).

Anyway, from what I have seen so far, Stardock has some top notch employees and I give them kudos for such an amazing product and for all the hard work that they pour into it. Furthermore, I would like to say that this Dev Journal is a really cool idea and is extremely interesting to read.

Also, on the topic of the Dev Journal, I have a question for the Stardock employees. It seems from reading this journal that GCII was written in C/C++ (as most games are from what I can tell), just wondering which one (if it is in fact C/C++)? Furthermore, was the AI written in the same language? I ask because having done an AI class we used Prolog for all of our AI stuff. While, IMHO, Prolog is a god awful language that should be locked away and never let out, it is still amazing language for AI. Well at least once you get your head around the recursion (which by itself is not bad, but when _everything_ has to be recursive

). Anyways, I noticed that an AI program written in Prolog was only a few lines long, while the C or C++ version would be at least 100X longer. So HUGE kudos on the AI as well

Walldorf2000

on May 04, 2006

I may be able to put up an unofficial test build with my changes later today as a link.

I would be pleased to help you to make it rock solid.

I have a nice test set of PCs on my hand. From a Athlon 500 with 256 MB, a 128 MB ATI card and Win98SE to a Athlon 64 3000+ with an Nvidia card and a Pentium IV 3200 with onboard Intel. The latter have 2GB each and XP Professional.

Any proposal what we should look for or what especially we should test?

Vandenburg

on May 04, 2006

I may be able to put up an unofficial test build with my changes later today as a link.

Would love that.

On a sidenote: Ignore that flamebait the other person laid. It's not worth it. Let your loyal followers and the Drengin deal with him.

CariElf

on May 04, 2006

No offense to Frogboy intended, but he hogs the spotlight just a wee bit.

Heh. It's not really his fault. I used to be a lot more active on the forums, as anyone who's been around since GalCiv1 can tell you. This time around, though, I've been feeling more whelmed by sheer numbers of threads on the forums, support e-mails, etc. I end up doing a lot of the second level support on our games, which in itself I don't mind, but I also feel like I'm not getting enough done as a programmer. So I've been letting the dev journals slide.

Also, on the topic of the Dev Journal, I have a question for the Stardock employees. It seems from reading this journal that GCII was written in C/C++ (as most games are from what I can tell), just wondering which one (if it is in fact C/C++)? Furthermore, was the AI written in the same language? I ask because having done an AI class we used Prolog for all of our AI stuff. While, IMHO, Prolog is a god awful language that should be locked away and never let out, it is still amazing language for AI. Well at least once you get your head around the recursion (which by itself is not bad, but when _everything_ has to be recursive ). Anyways, I noticed that an AI program written in Prolog was only a few lines long, while the C or C++ version would be at least 100X longer. So HUGE kudos on the AI as well

We use mostly C++ but there is still some C code in the game. Is Prolog a functional language? I can't remember from my Comparative Programming Languages classes because we never actually used Prolog or Lisp. We spent most of our time on functional programming, which drove me up the wall.

We very rarely write recursive code in our games because of the overhead and possible stack overflows. I no longer have a grudge against it like I did in college, but unless it really is more efficient than doing it iteratively, I see no reason to use it. There's no point in writing recursive code for the sake of recursion; elegant code is often unreadable and makes it hard to debug and maintaint.

Vulcannis

on May 04, 2006

Prolog is a logic programming language, not functional. It's pretty far from cutting edge now days, but still useful in it's domain. It's aged better than C/C++, IMO. I doubt it would be any use for the AI in galciv though... it's generally easier to turn a strategy into say C code than a series of logic statements, at least for any game more complicated than tic tac toe.

I'm always amused at the differences in opinion about programming between game developers and other developer groups, such as business devs or language geeks. Recursion is a great example; most game devs seem to hate or at least avoid it, I think because the languages they tend to use (C,C++,Python) have absolutely awful support for it. Half the business devs I work with would barely grok what it is, and the language people tend to view it as just another tool in the box, albiet probably a more fundamental one than say iteration.

Welcome Guest! Please take the time to register with us.
There are many great features available to you once you register, including:

Richer content, access to many features that are disabled for guests like commenting on the forums.
Access to a great community, with a massive database of many, many areas of interest.
Access to contests & subscription offers like exclusive emails.
It's simple, and FREE!