the Sim Settlements forums!

Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!

Understanding CTDs in The Triangle of Death (Sanctuary, Abernathy Farm, and Red Rocket)

Status
Not open for further replies.
Acknowledgements


I would like to thank the following individuals for contributing to, influencing, or otherwise informing this guide, and everyone who replied to the original thread and provided feedback:

  • SmkViper
  • Sclerocephalus
  • GA_Darkerside
  • damanding
  • wim95
  • GamerPoets
  • Blick Mondoo
  • Yagisan
  • woodfuzzy
Original contributions have been slightly re-worded to fit the overall tone and flow of the article, but hopefully the spirit and details of the information were not lost. Any errors or omissions are my own.


- snarkywriter



Chapter 1 – A Summary of the Problem


Introducing The Triangle of Death

One of the more common areas of the game to encounter CTDs (crashes to desktop) and other gameplay issues when using Sim Settlements is the area of Sanctuary, Abernathy Farm, and Red Rocket, commonly referred to around here as The Triangle of Death. In order to understand why these crashes are occurring, we need to look at some of the core mechanisms Fallout 4’s engine uses to create the world around your character, how these core mechanisms are commonly circumvented when we mod our games, and what effects this can have on game stability.

First, we need to talk about triangles.


Triangle Rasterization and the Settlement Build Limit

Fallout 4 uses what's called Triangle Rasterization to render objects on the screen. Everything you see in the game is made up of a bunch of tiny connected triangles. The engine uses triangles because they are simpler to draw than other shapes, and in terms of geometry, every shape can be split (roughly) into triangles, but a triangle can only be split into other triangles. The settlement build limit measures the number of triangles the engine is trying to render in the settlement. Our computers, and the engine itself, can only render so many triangles in a given location at a given time AND still run all of the scripted functions it needs at the same time. Unfortunately, some of the tricks developers use to make more complicated scenery in games, like occlusion planes (which prevent anything behind them from rendering, thus saving precious processing power), cannot be used in free-built settlements and RotC cities.

However, players and mod authors quickly figured out ways to trick, bypass, and outright disable the settlement build limit, and it is a common practice to do so for people who build settlements extensively. With good reason, too, since the build limits on most cells fill almost immediately if you do any kind of building that involves decoration or custom objects. When the RotC city plans were created, the designers continued this practice and, in nearly all cases, radically exceeded the settlement build limits of the settlements the cities were housed in. This is particularly a problem in the area of Sanctuary, Abernathy Farm, and Red Rocket. Two of these three settlements (Sanctuary and Abernathy Farm) are large, and their vanilla build limits are much smaller than their physical sizes should be able to support. Thus, it is very tempting, and often necessary, to circumvent their budgets to get a logical, good looking, and good feeling build result.


uGrids to Load and uExterior Cell Buffer

The entire Fallout 4 map is divided into what are called Cells. To understand them and why they matter, you need to be aware of the settings "uGrids to Load", which is a game setting that tells the engine how many cells to fully render around the cell your character is in, and "uExterior Cell Buffer", which is a game setting that tells the engine how many cells to keep loaded in memory at any given time. The default values are uGrids to Load = 5 and uExterior Cell Buffer = 36. Whenever you pass a cell boundary, the game attempts to de-render/dump from memory this overlapping area and moves the loaded/rendered grid to center it on the new cell you are in.

The problem is that, standing in Sanctuary, an up to 36 cell area around your character is being loaded into memory, and everything contained in the 5x5 grid of cells centered around you is actually being rendered in-game. Depending on what direction you entered Sanctuary from, Abernathy Farm and Red Rocket are mostly within those ranges, and if the build limits have been exceeded in those locations as well... hopefully you see where this is going.

It's worse when you are moving, and worst when you are sprinting/riding a vertibird, as you are forcing the game to load, render, unload, and de-render that 5x5 cell area around your character pretty quickly. Also, these settlements themselves include multiple cells within the buildable areas, meaning moving around in them can cause you to change cells and trigger new load/render calculations. Add in the lack of performance tweaks that are necessarily excluded to allow players/RotC to build objects into the game world, all of the script calculations necessary to run the settlement systems in the background, the settler AI, and any other game scripts that are running in the background, and you can quickly cripple even the most capable computer.


Cell Borders and Moving in Settlements: Example 1

This is an example of the Fallout 4 cell map:

fallout-4-map-grid.jpg


Source: http://www.monkeymods.com/fallout-4...orials/fallout-4-map-grid-refrance-fallout-4/


In this example, you are standing in the cell where Carla idles in front of the player's old house in Sanctuary. Depending on how you entered Sanctuary, a good bit of the area of the image could be part of the uExterior Cell Buffer area, and nearly the whole island is within the uGrids to Load area:

uGrids Example v2018.8.19.jpg



Now, let’s simulate you running to the Sanctuary end of the bridge:

uGrids Example pt 2 v2018.8.19.jpg



Notice how the uGrids to Load area around you, which again is the area the game is graphically rendering, now encompasses most of Red Rocket, and has started to brush against the edge of Abernathy Farm, significantly increasing the number of triangles the game needs to try to render, as well as likely starting a number of script function calls which rely on cell load to trigger their firing (we'll cover this part in a moment).


Precombined Meshes and Previsualization Data (contributed by damanding, wim95, and GamerPoets)

Scrap Everything, which I love to death, exacerbates this problem greatly by removing the precombined meshes in the settlement cells, which normally function vaguely like "compressed" versions of the normal textures (that isn't entirely accurate, and I'm over-simplifying) and require less memory to load and then render graphically. With them off, ALL of the objects are rendered separately, adding a truly staggering number of things for the game to render, increasing the problem a lot.

Mod author damanding has written a guide on the Nexus forums which covers the technical details behind how the precombined mesh and previsualization systems work, available here: https://forums.nexusmods.com/index....ptimization-and-performance-systems-explained if you would like to know more.

There is also a short YouTube video by Michael at GamerPoets which gives a user friendly explanation the precombined mesh and previsualization systems, available here:
To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.
.

Mod author wim95 has created a test patch which re-generates the precombined meshes in Sanctuary, Abernathy Farm, and Red Rocket for users who are using scrapping mods like Scrap Everything, available here: https://simsettlements.com/site/ind...ata-in-triangle-of-death-with-scrap-mod.6923/ if you would like to test it.

(Continued below)
 
Cell Loading and Updating Settlement Information: Example 2

Additionally, this is where issues with settlement resource calculation come from in large, multi-cell settlements (aka. the infamous “Pip-Boy Resource Display Bug”). Let’s say you are running toward Sanctuary from Concord on the road that passes Red Rocket. You built a building full of generators in the empty lot on the northern side of the cul-de-sac, and ran power from it to a tower beside the main bridge that is covered in Laser Turrets:

uGrids Example pt 3 v2018.8.19.jpg



As you near Sanctuary and it begins to load into memory and render, you decide to check your Pip-Boy. The workshop scripts start to update. The cell our generators are in has not rendered, so unfortunately they don't exist as far as the workshop script calculations are concerned. Thus, your laser turrets all appear to be unpowered (because at the time of calculation, they are), and your defense value suddenly and inexplicably tanks in your Pip-Boy. The same thing can happen for the bed count, food count, etc. These counts happen before the happiness calculation is updated for obvious reasons, and thus the happiness begins to drop for no good reason. If you turn around and head back out, the workshop calculation saves like this, and your happiness will continue to drop/your settlement will be considered vulnerable to attack until you return to the settlement and let the scripts fully update.


It's Not Just the Triangle of Death (contributed by damanding)

For this post, we used the Triangle of Death to illustrate some of the core issues we as players and mod users encounter when developing settlements. Nearly everyone who plays Fallout 4 and engages with the settlement system encounters some part of the Triangle of Death within minutes of leaving Vault 111 for the first time. Heck, the game is basically designed to encourage players to start building in Sanctuary and Red Rocket almost immediately. Thus, it's a great example that most players can relate to.

Unfortunately, these issues are not exclusive to the Triangle of Death:
  • Any settlement that has a border within 5 cells of a path that players commonly travel can experience the “Resource Display Bug”.
  • Any settlement that has a border within 5 cells of another settlement’s border can set off rendering and script based issues/CTDs.

The Contribution of Scripts and Scripted Objects


So far, we’ve focused on the graphical rendering part of the equation. We also need to talk a little bit about how scripts contribute to the issue.

According to Sclerocephalus, one of the Unofficial Patch team coders, once you exceed a certain number of scripted objects between all of your settlements (this includes settlers, idle markers, turrets, Sim Settlements plots, beds, radios, everything that produces resources, etc.) you significantly increase the likelihood of choking out WorkshopParentScript, the core script that runs the settlement system in Fallout 4. See this thread for the technical details: https://afkmods.iguanadons.net/index.php?/topic/4462-fo4-unlimited-settlementssettlers-a-caveat-for-mod-users/& (Link no longer works - removed from external site)

They recommend never exceeding the vanilla max of 21 settlers in one settlement and minimizing the number of scripted settlement objects you create at any settlement, even if you use the UFO4 Patch version of WorkshopParentScript (which is different from the vanilla game’s script; luckily, Kinggath designs Sim Settlements to work with either the vanilla or the UFO4 Patch versions of the settlement workshop scripts, so it's safe to use both). Script lag on the various workshop script calculations exacerbates all of these problems and increases how often it can occur. There are a lot of areas in the Sanctuary, Abernathy Farm, and Red Rocket triangle where a whole lot of scripted objects and settlers can be rendered and contributing to the overall script load.


It’s Not a Bug, It’s a Feature…

Finally, it's important to note here that all of this is vanilla behavior that is based on fundamental engine mechanics. There aren't fixes for this "bug" since it is really just an edge-case artifact of how the fundamental systems interact and overlap. The problem for us is that the way we as users and authors like to mod settlements results in these edge-cases occurring with enough frequency to be, at best, irritating, and at worst, game-breaking.
 
Last edited by a moderator:
Chapter 2 – Solving the Problem

Call the Brute Squad!

Most of us have, at some point, thought to ourselves “wait a minute! I can brute force POWER my way through these problems! If I could only upgrade my computer, I could make Fallout 4 do as much as I want!” Along the same lines, users encounter issues with Fallout 4 and frequently make statements that are some variation of “but my computer is a BEAST. I have an overclocked Intel Core i9-7980XE CPU with liquid nitrogen cooling, 2TB of RAM, SLI’d Titan V graphics cards, and a 2TB 970 EVO SSD dedicated just to Fallout 4 and my mods! Why am I still CTDing?!?” (I’m being hyperbolic, but you get the point.)

Unfortunately, your computer’s specs only matter to a point, and that point isn’t much higher than Fallout 4’s recommended system specs. The “bottleneck”, a phrase I hear in gaming circles frequently, isn’t your computer. It’s Fallout 4.

This is about to get a bit technical, but to understand why our computer’s specs only matter to a point, we need to peek under the hood of Fallout 4 and map some of its major parts. Hang on to your hard hats, and please keep your hands and feet inside the vehicle at all times!

Bring on the technobabble

As you may have heard, Fallout 4 runs on what is called the Creation Engine. Papyrus VM, the scripting system for the Creation Engine, is only one of many different sub-systems that the Creation Engine has to allocate your computer’s resources to in order to run the game. The Creation Engine allocates most of the resources it has available to rendering the graphics of the game. This makes sense, because we all want great graphics, right? It’s usually the first and often the most important thing most gamers rate a game on.

The rest of your computer’s resources are split between all of the other sub-systems, of which the Papyrus VM is but one, and all of this is tied to your frame rate (the rate at which Fallout 4 is updating the still-frame images it is rendering). The Creation Engine uses the frame rate as a sort of “timing belt” to synchronize all of the sub-systems it is running. Because the sub-systems have very different capabilities in terms of how quickly they can execute their different roles, Creation Engine was designed with a target and maximum frame rate of 60 frames-per-second for computers, and 30 frames-per-second for consoles. If Fallout 4 is running at the target/maximum 60 frames per second, one frame is being rendered every 16.67 milliseconds. (If Fallout 4 is running at 30 frames per second, one frame is being rendered every 33.34 milliseconds.)

While Papyrus VM does not affect your frame rate, your frame rate absolutely affects Papyrus VM (source: https://www.creationkit.com/fallout4/index.php?title=Papyrus_FAQs ).

We Need to go Deeper...

The Sim Settlements Wiki includes two setting tweaks that we need to talk about to understand how the Creation Engine allocates resources to the Papyrus VM system during this miniscule timeframe: “fUpdateBudgetMS” and “fExtraTaskletBudgetMS” (https://simsettlements.com/web/wiki/index.php?title=Performance). These two settings control how many milliseconds per frame the Creation Engine gives Papyrus VM to run scripts (see here for the technical details: https://www.creationkit.com/fallout4/index.php?title=INI_Settings_(Papyrus) ). The default value for these settings is 1.2 (or 1.2 milliseconds), and the Performance guide recommends doubling them to 2.4 (or 2.4 milliseconds). Using the Performance guide settings, the Creation Engine gives Papyrus VM 2.4 milliseconds to run scripts every time Fallout 4 renders a new frame on-screen.

Luckily, Papyrus VM is a multi-threaded scripting language (see here for the technical details: https://www.creationkit.com/index.php?title=Threading_Notes_(Papyrus) ), which means that it can attempt to process multiple scripts concurrently during this tiny window of time. Most scripts, like WorkshopParentScript, are part of giant spider webs of interconnected child scripts which generate and update different variables and values that are then passed around the network. The parent scripts then bring all of these variables together to do something in the game. A system known as the call stack is used to map out this network of interconnections.

This brings us to the next set of setting tweaks the Sim Settlements Wiki includes that we need to talk about: “iMinMemoryPageSize”, “iMaxMemoryPageSize”, and “iMaxAllocatedMemoryBytes”. These setting control how much memory, in bytes, Papyrus VM can allocate to organizing and running all of the scripts in the call stack. The default values for these settings are iMinMemoryPageSize = 128 (or 128 bytes), iMaxMemoryPageSize = 512 (or 512 bytes), and iMaxAllocatedMemoryBytes = 153600 (or 156 kilobytes). The Performance guide recommends doubling these values to iMinMemoryPageSize=256 (or 256 bytes), iMaxMemoryPageSize=1024 (or 1024 bytes), and iMaxAllocatedMemoryBytes=307200 (or 312 kilobytes).

If Papyrus VM cannot complete a script in the 2.4 milliseconds it has during each frame, it pushes it to the next frame. According to SMKViper, the Bethesda employee in charge of Papyrus, Papyrus VM will keep doing this until either (a) the script completes or (b) Fallout 4/your computer/your XBox runs out of available memory. There is no such thing as “dropping a script”. The only thing that can happen is the scripting engine getting overloaded (source: https://www.creationkit.com/fallout4/index.php?title=Papyrus_FAQs ) and eating up memory.

I AM THE BRUTE SQUAD. (contributed by Yagisan)

Unfortunately, sheer computational power won't do it because the memory allocation sub-system was designed to run on both consoles and PCs. The memory allocation sub-system has to be able to work for Xbox and Playstation, as well as a PC. Each of the different consoles and PC systems have different methods for determining how system resources are distributed. The Creation Engine's memory allocation sub-system appears to have been designed with a standard in mind that can accommodate the majority of the systems it is meant to run on.

Remember, we have up to 2.4ms to do as much Papyrus VM work as we can. We can't push the frame rate above 60fps, due to its controlling of papyrus timings. However - we can partially brute force it. A faster CPU (single core performance here) can push through more of the Papyrus scripts in that 2.4ms it has to process. Multiple cores aren't going to help this particular performance aspect (but they do help others).

So, can you brute force it? Well, the answer is a "yes, but…" Yes, but if you are doing this, you'll get diminishing returns.

Now, what happens if you overload the Papyrus VM? It starts dumping to memory. You probably won't notice until it hits swap - then it will crash/grind to a halt. But by this time, you are beyond what your CPU can handle in that 2.4ms. The occasional overload is OK. It will catch up in a less busy time. But constant overloads? That's when you need to reduce your mod list and cut down on your 60 settler magnum opus, because the reality of having a very powerful computer is that you have a little bit more "wriggle room" to push the system before you'll also crash.

Lock your windows. Lock your doors. Lock your scripts. Lock your frame rate

Once a thread starts running a parent or child script, it “locks” that script until it is done processing, which prevents other scripts from accessing it and reading its variables. If the scripts are massive rats-nests of hundreds of different child scripts that all have to generate/update their variables every time (like WorkshopParentScript, unfortunately for us), and they use functions (these are the algebra/calculus equations that the Papyrus scripting language represents) that get the job done but run slowly (remember, slowly here is a more than a millisecond or so), this locking means that the parent script could take literally minutes to run. This means it could take many, many, many frames for Papyrus VM to eventually run the final parent script and perform the action you asked it to perform.

Frame rate is the key. Remember, Creation Engine was designed with a target and maximum frame rate of 60 frames-per-second for computers, and 30 frames-per-second for consoles.
  • No matter how powerful your computer, Creation Engine cannot give Papyrus VM more than 2.4 milliseconds (1.2 milliseconds for consoles and users who do not modify their .ini file) out of every frame to run scripts.

If you are running too many scripts, they will eventually start to back up, which increases memory usage, which crowds out other things like the rendering sub-system, which slows down the framerate, which backs things up even more, until this cycle eventually chokes out the Creation Engine and, boom, crash to desktop. This is even worse if you are asking the rendering engine to render a bunch of high-resolution (i.e., lots of triangles), complex graphics.

Brute force only buys us time.

(Continued below)
 
Last edited:
But, if I uncap my framerate… (contributed by woodfuzzy)

At least a few of you are probably thinking: “Wait a minute! If I just uncap my framerate, I can get 144 frames per second, and then everything will be faster!

Well, sort of.

According to SmkViper (here: http://forums.bethsoft.com/topic/14...lag-going-over-10000-ms-sometimes/?p=23340131 ), raising the frame rate actually reduces the amount of time the scripting engine has to process everything it needs to process in a single frame. So, at 30 fps, a new frame is being rendered once every 33.34ms. At 60fps, a frame is rendered every 16.67ms. At 144fps, a frame is rendered every 6.9ms. Remember, the default for “fUpdateBudgetMS” and “fExtraTaskletBudgetMS” (the number of milliseconds per frame Papyrus VM gets to work) is 1.2ms (2.4ms if you use the Performance values).

You’re asking for trouble if you are using enough scripted mods to require the full processing time of 1.2ms (or 2.4ms) per frame, but a frame only lasts 6.9ms. You are giving the rest of the game’s systems only 5.7ms per frame (or 4.5ms per frame with the performance settings) to perform all other functions the game needs to operate.

Granted, the game will run with a framerate higher than 60fps. But, if you let it go for long enough, you'll CTD at some point because letting the game render unfettered effects the time scale the scripts use, which causes scripts to run when they shouldn't, NPC's to move faster, animations to run faster, etc.

Added to this, the game engine has built in errors that are normally dealt with by the engine as it runs, such that you never notice. But, unlocking the FPS causes the errors to pile up so fast the engine can't deal with them... boom CTD.

You can achieve the same effect by setting the game’s timescale to 120 using console commands (the default is 20). Within just a few minutes it will crash. Even setting it to 30 will cause it to crash, although it will take a lot longer. The Papryus VM system is very fragile when it comes to timing. It can be pushed a little… but one good shove and it will fail.


An Analogy

Imagine Fallout 4 and your computer as a car. The engine is your computer. Everything else, the body, interior, frame, steering system, suspension, tires, power transfer system, braking system, etc., is Fallout 4.

Fallout 4 is a mid-sized, 4-door sedan. Every one of its systems was designed to run best with the factory in-line 2.4L 4-cylinder engine (in this analogy, the system specs for an Xbox One).

Your computer is the 8.0L quad-turbocharged W16-cylinder engine off a Bugatti Veyron.

You take your mid-sized sedan, and shoe-horn in the engine from a Bugatti Veyron. It'll drive, handle, brake, steer, accelerate, and idle just like a Veyron, right?

Of course it won’t. Don’t get me wrong; it’ll go faster than the 4-cylinder will… until everything else fails because it’s being worked well outside its tolerances.


But, what CAN we do?

There are a couple of basic things we can do to minimize these problems:

1. Be very selective of what settlements you choose to build as a Rise of the Commonwealth city plan. DO NOT try to build all three settlements in the Triangle of Death in the same playthrough, and DEFINITELY don’t try to build all three from scratch at the same time. The same goes for Warwick Homestead, Spectacle Island (while vanilla SS does not include a city plan for this location, plans are available in other mods), and The Castle.

2. When you enter a settlement, walk to the center of the settlement and give it a few real-time minutes to run update scripts BEFORE you try to open the workshop menu and build.

3. Try not to pack too many scripted things into settlements that are close to each other. This includes keeping the populations of the settlements you build yourself modest.

4. If you can, avoid using mods that (a) expand settlement borders and (b) disable precombined meshes.

5. Beyond the recommended .ini tweaks from the Sim Settlements Wiki’s performance guide, avoid doing things like (a) changing the timescale, (b) uncapping your framerate, (c) changing other .ini settings, (d) running unnecessary programs in the background on your computer while playing, (e) otherwise setting yourself up to crash.
 
Last edited:
Status
Not open for further replies.
Top