The VIS stage
Taken from SmallPileOfGibs' q3map explanation.
The VIS stage of q3map2 creates the table of Potentially Visible Sets (PVS) for the Portals in a map. As the player's viewpoint moves around a FULLY vis'd BSP, different areas become visible or hidden, depending on the area the viewpoint is in.
How q3map2 creates the PVS
Every Portal in the .prt file generated by the BSP stage is checked against every other Portal for visibility. Portal 1 is able to view Portal 2 if a straight line can be drawn between any part of Portal 1 and Portal 2 without structural brushes getting in the way. Every Portal then gets another list of the other Portals which can be seen from it. This information is the Potentially Visible Set, which is stored in the PVS Table. It's possible to see the PVS in-game by using r_lockpvs 1.
With the -fast option enabled at the compiling moment, this PVS Table is not created, leaving every Portal visible to other portals as well. This accelerates the VIS process but is prone to cause framerate problems in-game, so it's not recommended.
How the Portals affect visibility
Every Leaf Node has one or more Portals, unless there is only one node in the BSP Tree. When the player's viewpoint is in a Leaf Node, every object (Brush faces, Bezier patches, Entities) in every other Leaf Node visible from that Leaf Node is drawn. Likewise, objects can be in more than one Leaf Node at a time.
Brushes with shaders with the structural property such as common/caulk or common/hint block visibility between the contents of two Leaf Nodes. Curves, brushes with the detail parameter set and Entities won't block visibility. The visibility between two areas is blocked when both areas are completely hidden from each other. Generally, the more areas being seen from the Leaf Node the viewpoint is on, the more objects are drawn.
As such, when working on performance, one must bear in mind how many objects are being drawn from a Leaf Node, as the Leaf Nodes doesn't always form where the mapper wants then to, and can be part of the PVS even when structural brushes apparently intervene. This is why the OA engine could be drawing areas which players can't see, because the engine thinks they can.
The first cuts are based on the blocks which are 1024x1024x1024 cubes. Next, it picks the first chunk and checks if that space is convex. If not, then it picks a brush face in that space and splits the space in two along the plane of the brush face. If the space was concave and was split in two to make new Leaf Nodes, then q3map2 has to check if the two new Leaf Nodes created to see if they're still convex, and split them again. This process continues until all the Leaf Nodes are convex. Then it moves to the next chunk and repeats the process. Eventually, the whole map is splitted up into convex spaces.
One way to reduce such number of objects being drawn is to reduce the number of structural planes by making every brush which doesn't block visibility into a Detail brush. Marking brushes as "detail" will stop them from affecting the BSP Tree. This leaves a low number of portals, thus speeding up the VIS process, but this doesn't always solve the visibility problem, as most stuff is still being rendered even when the player can't see it. Enter the Hint Brushes.
Hint brushes are brushes placed in maps to give "hints" to the compiler about where visibility portals should be placed, hence the name. They are invisible in OA (have the nodraw and nonsolid properties), but they are structural, so they will affect the BSP Tree and create more Leaf Nodes. This allows mappers to control where and how the Leaf Nodes are created. Hints can also intersect with other structural solid brushes or hints, creating multiple Leaf Nodes or isolating groups of them.
By using them in order to make axial cuts along planes shared by other structural brushes, it's possible to maximize the amount of area hidden by each VIS-blocking structural brush, minimizing the size and amount of visible Leaf Nodes.
Intelligent use of Hint and Detail brushes reduces r_speeds and vis time in almost any map, so it's not only recommendable, but mandatory, to plan your maps with this in mind, as redoing an inefficient map is a lot of work.
The use of hint brushes is a no-brainer: create a brush with common/skip in all sides, and common/hint in the face which will act as the portal.
Now, where should we use these brushes? Load the .prt file in both the 3D and 2D views, and filter everything sans World, Caulk, Structural, Entities (only if you've used a func_* entity), Translucent (because of the common/skip texture) and Hints (at this point we haven't placed a single brush, so if we're going to work with them...).
Now look for places where the visibility can be blocked. This is possible if, by drawing a line from a certain place, a certain area doesn't get renderized. Let's see an example.
The easiest way to think about hint brushes is by imaging virtual lines of sight. So here goes a practical example:
This place is having too many hidden tris being rendered.
The player isn't seeing that stair there, but the render is.
Many tris from this big area are being rendered too.
Placed hint brushes like this. That angled slice is matching a virtual line of sight of a player standing in one of the corridor's edges or the other.
The hint brushes forced q3map2 to generate visibility portals like this.
Now when a player is standing here, behind that portal, looking at the wall and not seeing the stairs that are up ahead...
This is the result: much fewer tris behind the wall being rendered.
An alternative line of sight.
Portal view, the new line of sight. But the older line is still there, because there is still a hint brush with an angled face following that line.
Now if the player stands in this place the render shouldn't see the stairs. In comparison to the older line of sight, the new one provides some extra few steps that the player can walk farther on without having those stairs rendered.
Optimizing hint brush placement
Suppose that you are in a place in your map that falls under the "many angles" case where there are many lines of sight. Where to place the hint brush? Which angle is optimal?
Think about "worst case", "best case" and "average case" scenarios. Rather than discussing theory behind that, let's see a practical example (remember, the 2D figure can be interpreted both in XY plane and in Z axis):
While player is at P1 or P2, A1 is not rendered. That is our "best case". The whole area is occluded.
While player is at P1, A1 is not rendered. While player is at P2, A2 is not rendered. That is our "average case". Part of the area is occluded and part is not.
While player is at P, A1 and A2 are not rendered. That's our "worst case". Tiny parts of the area are not rendered, while most of it, is. In fact, "best" and "worst" case are the same hint brush angle. The difference is just the player's position, whether "in front of" or "behind" the portal.
Combine all three cases for best results.
It's the same as the previous one but with two openings. Same hint brushes idea, just that there are more hint brushes this time.
Now what happens here?
Angled hint brushes on either extreme side can be placed because the cluster is large enough to cover many things. But angled hint brushes on P1 - P3 depends on how large the sections are, if they are large enough to occlude things, it's worth of placing them. If they are narrow, then axial only. If they are even more narrow, like pillars, then forget about hint brushes there.
To conclude, what about "incremental" line of sights? Check this following figure:
There are many many lines of sight, with one hint brush for each angle. Placing hint brushes like that is futile for two reasons: first, it'll take too much time to compile for too little gain; second, consider the player's size and average walking / running speed, if the space between each portal is too thin, the player itself will always be standing in two or more overly thin vis clusters. Plus, if the player can cross two or more clusters with just one footstep there won't be any noticiable gain in fps. If you really want occlusion like that, then forget about pre computed visibility and start thinking about a realtime, dynamic, solution.
Hint brush fail
Why does it fail here? (Because a straight line can be traced from cluster 1 to cluster 3 and vice-versa).
Why does it fail here? (think in 3D. If player is on cluster 1 or 3 and those cluster' height is higher than that yellow block, then they can see each other. The render can trace a straight line from above the yellow block. Sometimes q3map2 will automatically split the cluster along the Z axis and XY plane, but if it doesn't, you have to do it with a hint. In other words, this is a two in one case scenario, "L" shaped corridor in XY plane and "U" shaped corridor in Z axis.)
Why occluded surfaces are still rendered? (Because it's either 100% occluded or not, there is no such thing as "half occlusion". If one pixel of a triangle falls under a visible cluster, then the whole thing is rendered. Hint brushes do not split large surfaces in smaller ones, q3map2 doesn't do that. Manually splitting large surfaces in smaller ones for the sake of vis is not something you wanna waste time on doing.)
- The first consideration to bear in mind is that HBs are workarounds for compiler limitations and should be used with care. While a well placed hint can save a lot of rendering and reduce vis size, they will always increase the bsp size. Also, a badly placed or redundant hint brush will lower performance.
- HBs don't need to be blocks which fills a whole area. The hint brush faces facing the walls and floors are automatically culled.
- HBs can be thin, having common/hint in just one face and common/skip in the rest; or thick, having all the faces with the Hint texture. This last method isn't recommendable, as every face with the Hint texture which isn't touching a brush will make a cut in the BSP Tree.
- HBs don't need to be one continuous surface, it's possible to split the brush in more brushes to adjust to the surrounding geometry; as long as the smaller pieces have the hint face lying on the same plane, the compiler should "see" that surface as one portal.
- HBs can go through detail and structural brushes, but sometimes it's better to not make the hint brush go through structural brushes, because the portal generated by it might "invade" spaces where it shouldn't.
- SmallPileOfGibs' q3map explanation
- WeMakeMaps.com Hint Brushes Part 1
- The size of these cubes can be changed by modifying the "_blocksize" key with another power of 2 (512, 2048...) in the Worldspawn entity.
- Another way of creating them is by texturing all the brushes with the common/hint texture, but this may create portals where they aren't needed.
- WeMakeMaps.com Hint Brushes: Part 1 - Part 2 - Part 3 (internet archive)
- Detail and hint brushes: Part 1 - Part 2 (internet archive)
- Understanding Vis and Hint Brushes by jex on tremmapping.pbworks.com
- Alternate location: linradiant.intron-trans.hu
- Hint brush tutorial by djbob on Surfacegroup.org
- Alternate location: wolfensteinx.com/surface (internet archive)
- Potentially visible set on Wikipedia