Multithreaded map compiling

It is possible to take advantage of modern multi-core processors to compile maps much faster by using multithreading.

Do you need this?
Well, map compiling speed can really be improved a lot by it.

If you are using Linux, map compiling uses a single thread by default, and you have to edit your "build/bsp" menu options in order to specify the number of threads to use. This can really speed up map compiling a lot!
Please notice: "Linux not automatically setting the number of threads" info is taken from some old Radiant/q3map2 documentation. But we know about mappers for which multithread compiling happens automatically under Linux. There are many different Linux distros and various q3map2 forks out there, and both are updated over time. This means nowadays in some or even most cases, your linux q3map2 might actually be using multithread by default!

If you are using Windows, map compiling will use as many threads as your cores by default and usually you don't need to tweak it, unless you want to use fewer cores to have the computer more responsive for doing other tasks while compiling.
If you wish to use multithread for .AAS (bot navigation) file creation, then you will have have to specify the number of threads manually, even on Windows.

Overview

Note: if you already know what how many cores your PC has got and you already decided how many threads you wish to use, you can skip this section.

First of all, multithreading can only help you if your machine has got multiple "cores", at least "logical" ones. However, probably since around 2010, all PC processors have multiple "cores".

Expand this box for infos about how to find out the number of cores of your machine

A quick way to see if your machine has got multiple cores (or multiple processors):

• Microsoft Windows: to take a look to the "Performance" tab of the "Task manager" tool (from Ctrl+Alt+Del).
  • In Windows 8/10 Task Manager, under "Performance" -> "CPU" screen, it lists the number of physical processors, cores and logical processors as simple text right under the graph.
  • Otherwise, you can check wheter the "CPU utilization history" graph can be divided into two or more graphs... enabling a certain option will show as many graphs as your total number of (physical or logical) cores, so you can "count" there:
    • Up to Windows 7, under "View" -> "CPU History" menu from Task manager "Perfomance" tab, you can find two options: "One Graph, All CPUs" and "One Graph per CPU": the second one is the one required for this test, and is enabled by default. In case you have "One Graph, ALL CPUs" checked instead, you will NOT be able to find out the number of cores, unless you modify the option.
    • If you are using Windows 8/10, you will not find the "CPU history" item under the "View" menu: you have to right-click on the CPU graph and choose "Change graph to" -> "Logical processors" instead (by default, a single graph is showed there!).
• GNU/Linux: run the "top" command from a Linux "shell" (command line interface). After pressing the "1" key, in the upper part of the screen, the "Cpu(s)" line should change into multiple lines ("Cpu0", "Cpu1", "Cpu2"...), one for each core (physical or logical).

If you have got more than one core, you may find useful to compile maps (q3map/q3map2) using more threads, in other words splitting the process into separate sections that can run at the same time on multi-core systems. This way, you should see higher CPU load in Task Manager/top while compiling, and you should be able to compile maps much faster than using a single thread, because each thread should work on a separate core (with a single thread, only one core would be used). So, choose the number of threads (1, 2, 3, 4, 5, 6...) depending from how many cores you have got and how many of them you want to use for this purpose: using all available cores should give you faster compiling. On the other hand, leaving at least one physical core "free" should allow you to do not experience slowdowns using the computer do to other things (e.g. playing a game or surfing the web) while you are compiling a map; it's your decision. You may wish to create various custom pre-sets using different numbers of threads, and choose each time which one to use depending from how much processing power you wish to have free for other applications.

How to setup multithreaded compiling

Q3MAP (map compiler that comes with the original Q3Radiant) and Q3MAP2 (map compiler that comes with GTK Radiant, with some extra features) are command line tools. The map editor launches q3map (or q3map2) with various command-line parameters, depending from the Bsp option you choose from the menu. Q3MAP and Q3MAP2 support the "-threads n" parameter to tell how many threads they have to use ("n" is the number of threads, e.g. 1, 2, etc.).

• While running on Microsoft Windows OS, Q3MAP2 auto detects the number of total cores (physical + logical) your machine has, and automatically selects the number of threads to use accordingly. However you can use the "-threads" parameter to manually force the number of threads to use.
  
• While running the compiler on GNU/Linux OS, instead, you have to manually specify the number of threads to use. Or at least, this is what some old radiant/q3map2 documentation said. On your system it may work automatically, as we know least for some users it does.
  
• Also the old Q3MAP can automatically use a thread for each core it detects on Windows, if you don't specify the "-threads" parameter... but unfortunately some options in Q3Radiant "Bsp" menu do actually contain "-threads 1" by default, and so you will have to manually modify them if you want to use multiple threads.
• BSPC tool (used to create .aas files) does not autodetect the number of cores, not even on Windows. You have to specify the number of threads if you wish to use multiple cores. See BSPC section for more infos.

Expand this box for some infos about physical and logical cores.

Map compiler doesn't know the difference between complete processors, real cores and SMT logical cores (SMT stands for Simultaneous multithreading, and is referred by Intel as Hyper-threading (HT) technology, while also AMD has its own implementation of SMT). It'll assume that each "CPU" it finds is a complete CPU.

In reality, each physical processor can have one or more physical cores. Each core may have simultaneous multithreading feature, that would show it as two cores. So, a dual-core processor with SMT enabled may look like a four-processors machine, but the performances would be lower than a four-processors machine! While a dual-core CPU has got two physical cores that share some components, SMT technology shows an extra (logical) core by simply doubling few components in one physical core, thus it is not as efficient: depending from the different applications and environments, in a few cases SMT may even hurt performances; fortunately, it looks like this is not our case, and that you can actually take advantage of SMT.

There is no advantage in trying to force more threads than the number of total cores you have. While using all your physical dual (or quad, or more) cores should speed up a lot, the effect of SMT (logical) cores may vary; anyway it seems SMT (e.g. HyperThreading) is quite good for map compiling, and using all of your cores should give you better results. If you wish, you can do tests compiling the same map specifying different number of threads, comparing the time needed.

BSPC

In a nutshell: just add -threads <n> (change <n> with the number of threads you wish) to your BSPC command line. Done.

If you wish more detailed infos, continue reading this section.

BSPC (the tool that creates "bot navigation" .AAS files) supports multiple threads, but it does not auto-recognize the numer of threads to use (not even on Windows), and will use one thread by default. You can however specify the number of threads to use by adding the -threads <n> option, gaining compiling speed on multi-core machines. Just an example, if you lauch BSPC from a batch file that executes bspc -bsp2aas mymap.bsp, you can modify it to execute bspc -threads 2 -bsp2aas mymap.bsp to run it with two threads instead of one. However, the time adavantage it gives is not as evident as with map compiling: test it and then decide... probably you can leave it without that option and still live happy (especially in case you notice enabling it causes BSPC to crash more often in your machine; however, switching to a different version of BSPC may fix the issue and allow you to use multithreading without crashing, hence allowing you faster .aas compiling.).

You may also specify the -threads <n> option to the batch file you use to optimize the .AAS file you previously created -the operation that makes the .aas file smaller- (example: bspc -threads 2 -aasopt mymap.bsp), but that operation is already so quick you don't need multithreading at all for it. By the way, it is possible (and probably advisable) to create aas file and optimize its size with a single command, using a command line like bspc -grapplereach -optimize -bsp2aas mymap.bsp
Note: in this example, we also specified the "-grapplereach" option because it causes the "calulating reachability" phase to take some more time and you may take advantage of this to better compare BSPC performances when using different values for the "-threads" option. For your information, "-grapplereach" option is required for bots to use the grappling hook, if the server allows them to use it (bot_grapple 1). However in-game bot grapple support is incomplete and buggy since Q3A, and currently (OA 0.8.8) it has not been fixed yet.

NetRadiant

• 1) In NetRadiant 1.5 or NetRadiant Custom, go to "Build" -> "Customize..." menu.
• 2) There, click on the entry of the list that you want to modify, to highlight it.
• 3) Click on the line inside the "command line" section of the screen to highlight it, then click again to edit it.
• 4) Add -threads N (where N is the number of threads you wish to use -usually the total number of logical "cores" of your machine, fewer for a more responsive machine for other tasks during compiling-) at the end of the line, then confirm with ENTER.

Example: changing

[q3map2] -light -fast -super 2 -filter -bounce 8 "[MapFile]"

into

[q3map2] -light -fast -super 2 -filter -bounce 8 "[MapFile]" -threads 8

or (nicer)

[q3map2] -light -fast -super 2 -filter -bounce 8 -threads 8 "[MapFile]"

• 5) For the build options which include all passages (bsp, vis and light), add the option to all the three lines shown in the "command line" section. In other words, repeat step 4 as needed.
• 6) Repeat steps 2-5 for all the build options you wish to set the number of threads to use (e.g. those you are used to use).
• 7) Close the window with OK button.

Done.

You may instead wish to add new items in the "build" menu instead of modifying the existing ones, e.g. to have different options with different number of threads depending from how many cores you wish to leave free for other uses. In that case:

• 1) Go to Build -> Customize...
• 2) Click on the empty line at the bottom of the list to highlight it. Click again to edit it.
• 3) Type the name of your new entry (you can copy-paste from an existing name and then change something), then press ENTER.
• 4) Click on the first line (still emtpy) in the "command line" section to hightlight. Click again to edit.
• 5) Write the command line. You can of course copy-paste from an existing one and then change something. Specify the desired number of threads. Confirm with ENTER.
• 6) If you are making a build option which includes all passages: repeat steps 4 and 5 for the second and third line. Confirm with ENTER each time.
• 7) Close the window with OK button.

If you wish to delete an entry of the list, just highlight it and then press CANC. Or edit it, delete all characters to make it empty and then confirm with ENTER. Warning: there is no confirm request!

GTKRadiant

For GTK Radiant 1.5, see the section about NetRadiant, as it works the same.

For GTK Radiant 1.6, proceed as follows instead.

To edit an existing entry:

• 1) Go to File -> Project Settings.
• 2) Click on an entry of the "Menu commands" list, then click "Change" button.
• 3) Modify the Command field to add in the "-threads N" option, for each stage. Replace N with the desired number of threads, of course.

In this example, we modified a complete compile with the three passages (bsp, vis and light), so you can find three "-threads 4" highlighted in bold to show what we changed.

! "C:/GtkRadiant-1.6.6-20180422/x64/q3map2" -v # -game quake3 -fs_basepath "C:/Quake III Arena/" -meta -threads 4 $ && ! "C:/GtkRadiant-1.6.6-20180422/x64/q3map2" # -game quake3 -fs_basepath "C:/Quake III Arena/" -vis -saveprt -threads 4 $ && ! "C:/GtkRadiant-1.6.6-20180422/x64/q3map2" # -game quake3 -fs_basepath "C:/Quake III Arena/" -light -fast -patchshadows -samples 3 -bounce 8 -dirty -gamma 2 -compensate 4 -threads 4 $

As you can guess, after "&&" a new command starts, so q3map2 program is invoked three times within a single menu entry. Being just an example, don't mind the fact that there GTKRadiant was configured for Quake 3 Arena instead of OpenArena. And of course, the paths will point to your actual GTK Radiant and game folders.

• 4) Close with OK, OK.

Done.

If you prefer to create additional entries (like to have different pre-sets with different number of threads depending from your need to perform other tasks during compiling), you can use "Add..." button, then enter a "Menu text" and a "command" (which, as seen, can consist of three concatenated q3map2), then close with OK, OK. You can of course copy-paste the text from existing menu entries and then modify, instead of typing everything from scratch.

Warning: when you create a new menu entry, or rename an existing one, remember that its "name" ("Menu text") has to begin with "bsp" (e.g. "bspMyTest", "bsp_MyTest", "bsp MyTest"...). If the name will not comply with this rule, you will lose your changes!

It's possible to delete a menu entry using the apposite "Remove" button after selecting it.

Tip: you can enlarge the "Edit Command" window by dragging its borders with the mouse. However you can also copy-paste its content into a plain text editor (like notepad), make your changes there and then copy-paste again into the window, if you prefer.

Trivia: editing the bsp menu list through "project settings" window is how Q3Radiant should have worked, in case it didn't have a certain annoying bug with it.

Q3Map2Build

With Q3Map2Build (a small but useful tool that allows to compile a map -and optionally, also its .aas file- without launching a Radiant editor, with full control over compiling options), you can specify the number of threads to use by clicking on the "Options" buttons corresponding to each stage (BSP, VIS, LIGHT, BSPC). Such "Options" buttons are active only if you have chosen "Custom" for that stage. While in selected stage options window, just check "-threads" option and specify the desired number of threads in the apposite field, and click "Save".

After you set options for all stages, build (compile) the map.

Q3Radiant

Note: please read all this section before trying to apply it: as later said, a bug in the editor may mess up your compiling options even if you didn't actually change them! In short, to avoid that bug, you will need to manually tweak a file called "quake.qe4" with a text editor (but it's worth the effort, as unfortunately some menu options did have "-threads 1" by default).

Let's take the case we are using the old Q3Radiant editor, configured for the original Quake III Arena game, on Microsoft Windows.

In the "Bsp" menu, you can find the bsp_FullVis (light -extra) option. This is probably the one that takes more time to compile, used also for the "final" map, and unfortunately this menu option specifies "-threads 1" parameter, so it is the one we will tweak. You may notice great time benefits after the tweak.

Go to "File" -> "Project settings" menu. In the "Menu commands" section, select bsp_FullVis (light -extra) line and click "Change...".
In the window that opens, select all the text in the "Command" field, and copy it to your clipboard (CTRL+C or right button, copy), then click on "Cancel", to be sure we will not damage the original option.
Now click on "Add...", and paste from your clipborard into the "Command" field (CTRL+V or right button, paste). Modify from -threads 1 to -threads 2 (or other number, e.g. 3, 4, 5, 6...) in the two places where the "-threads" option appears, and optionally add a third one in the first part of the command if you like (as you can see, this menu entry invokes q3map program three times -a new command starts after "&&"-, and only two of them have the dreaded "-threads 1" option). Enter bsp_FullVis (light -extra) Multithread (or whatever you prefer, but anyway the name has to begin with bsp, and it should not contain the " character) in the "Menu text" field, then click OK.

You should have started from something similar to this (of course your Q3 installation path may vary):

! C:\PROGRA~1\QUAKEI~1\Tools\q3map $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -vis -threads 1 $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -light -extra -threads 1 $

And you should have got something like this (in case you want 2 threads):

! C:\PROGRA~1\QUAKEI~1\Tools\q3map -threads 2 $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -vis -threads 2 $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -light -extra -threads 2 $

Still in the "Project Settings" windows, select the newly created option and click "Change...". If everything is still as you set a minute ago, you are ready and you can go compiling your map... but if the line in the "Command filed" appears incomplete (where many options misteriously disappeared), e.g. like this:

! C:\PROGRA~1\QUAKEI~1\Tools\q3map $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map 

you enconountered a bug of the editor (this is another reason we suggest to never modify the original menu options) and thus you have to manually modify the file that contains the menu to fix the menu option (compiling would not run correctly without the missing part of the command line).

In this case, close Q3Radiant. Then go to your Quake III Arena installation folder (if you have multiple installations of the game, the one you use with Q3Radiant), then enter the "BASEQ3" folder, and the "scripts" folder in it. There, use a plain text editor (such as notepad) to edit the "quake.qe4" file. Locate the line that contains your newly created menu option (in our example, bsp_FullVis (light -extra) Multithread).
As you can guess, the format of a line is simply "menu text" "command". Correct the command line part, copy-pasting from the original menu option and changing the value in the two places of the line where the -threads option is. Save and close the file, and start Q3Radiant again.

At this point, if you select the newly created option from the Bsp menu, you should notice your map compile in less time.

Of course, you may have directly edited the quake.qe4 file and added a line (somewhere between the { and } lines) for the new menu option, if you prefer. Also in this case, remember that the name of the new menu option has to begin with bsp and should not contain the " character (that is instead used to separe the menu option name from the command line).

Some hints:

• Also considering the editor bug, you may want to make a backup copy of that quake.qe4 file before starting to make any changes to it through Q3Radiant or through text editor.
• In our example, we specified the number of threads we want. Anyway you can set q3map to automatically use the same number of threads as the total number of CPU cores Windows finds, by removing the "-threads n" option from the desired line in quake.qe4 file, in the two places of the line where it appears.
• You may also wish to edit quake.qe4 file with a text editor to quickly update all the paths in case you changed your Q3A install location. But note that the "Prefab path" string in "Q3Radiant Preferences" window is stored in some other file.

See also

• Mapping
• Mapping resources & tutorials
• Speed up visibility calculations