Real time shadows - the good news, and the bad news :P

Diedel · Post by **Diedel** » Tue Nov 28, 2006 7:33 am

The good news: I've got it to work.

The bad news: Virtually all standard D2 polygon models are so badly constructed that real time shadowing does not work with them. Same applies for Eagle's hires pyro.

I am currently having a custom ship I had built myself and have fixed in the course of testing my shadowing code that works, but I don't think it conforms to everybody's taste.

So if there are D2 addicts out there who have some spare time at their hands and like modelling, get the Descent 2 tools suite from PlanetDescent, fire up Polytron and fix the existing models or build new ones.

A brief guideline on models well built for shadowing:

Each submodel must have a closed hull, i.e. have polygons covering all sides of its body.
All normals (perpendiculars) on the polygons must point outside.
Don't use concave polygons, but split them up instead until they are all 'flat' (have all points on the same plane). Convex isn't recommended either.

Feel free to make them more detailled - today's hardware can handle a lot more then D2's simple models.

Diedel · Post by **Diedel** » Thu Nov 30, 2006 3:51 am

An update on poly models.

I have taken the original D2 Pyro and reworked it with Polytron. The original model is not just a mess, it's a nightmare. It has 284 polygons and 415 vertices (normally you have less vertices than polygons, as many polygons share vertices). The remodeled Pyro has 135 Polygons and 132 vertices.

Read more in my work log.

Chaos Death Saurer · Post by **Chaos Death Saurer** » Thu Nov 30, 2006 5:44 am

Can you keep a log of what models have and haven't been done? That way we don't get duplicate models and we can see how far along we are.

Diedel · Post by **Diedel** » Thu Nov 30, 2006 6:38 am

Sure. Currently I am still at work with the Pyro. I thought I could the saved polygons to spice it up a little. Pimp my pyro, heh.

DarkFlameWolf · Post by **DarkFlameWolf** » Thu Nov 30, 2006 6:46 am

yeah, at the moment, the shadows look icky.

They can also pass through walls into your view even when the robot is clear across the room with dozens of obstacles between you and it.

Diedel · Post by **Diedel** » Thu Nov 30, 2006 1:02 pm

The problem is that if the models aren't closed, the shadow volume isn't closed either. This will cause the shadow computation to fail, creating black artifacts. Basically, for each front wall of the shadow volume, there must be corresponding rear wall in front of an object in order for it not to be in shadow. It's like +1 + -1 = 0. If one of '1's is missing, you get shadows were they shouldn't be.

Chaos Death Saurer · Post by **Chaos Death Saurer** » Thu Nov 30, 2006 6:24 pm

You were right, Deidel: the models are a mess. I'll get cracking on the bosses this weekend.

Diedel · Post by **Diedel** » Thu Nov 30, 2006 6:58 pm

I have finished my replacement pyro and added support for replacement models to d2x-xl. See my work log for details.

CDS,

make sure that all sub-parts of your models are closed, even if they are stuck on some surface, like the spikes some bots have. You can stick polymodels into each other as long as they are closed though.

DarkFlameWolf · Post by **DarkFlameWolf** » Thu Nov 30, 2006 8:03 pm

hopefully you'll make an option in the menus to turn on/off the fixed models.

DCrazy · Post by **DCrazy** » Thu Nov 30, 2006 9:02 pm

DFW: Why? I'm pretty sure that Diedel is just re-drawing the exact same model with the appropriate number of vertices and faces.

Diedel · Post by **Diedel** » Fri Dec 01, 2006 3:16 am

If it only was so simple. I am using the original model as an offset, removing polygons until I have only the vertices left that make its shape, then add new polygons using these. I am not ripping away all superfluous polys at once, but I am doing this in little chunks. Polytron is a biotch to work with though. It has bugs, it is uncomfortable ... I have asked Heiko Herrmann for the source code. Hopefully he gives it to me so I can fix it.

Sirius · Post by **Sirius** » Fri Dec 01, 2006 5:42 pm

Polytron is also very limited and is completely unable to reproduce all but a handful of the Descent 2 robots in a manner that will actually work with the game. (It barely saves any of the BSP data because it isn't designed to handle it.)

OTOH, if you're using Z-buffering you might not notice that.

DarkFlameWolf · Post by **DarkFlameWolf** » Fri Dec 01, 2006 7:00 pm

you'd think someone would have made a better program by now to do those.

Chaos Death Saurer · Post by **Chaos Death Saurer** » Fri Dec 01, 2006 7:07 pm

I would think that.

Diedel · Post by **Diedel** » Fri Dec 01, 2006 7:24 pm

Well ... it's good enough to fix the D2 models.

So far I have fixed or checked the Guidebot, IT Droid (actually this one was alright), P.E.S.T. and P.I.G. Will do the Thief asap, and that will be all bots for D2 level 1. They are already included in d2x-xl.hog.

I will methodically proceed level by level of D2:Counterstrike.

CDS, if you want to do the bosses, I'll leave them for you. Or you start working down from the last level, and I will include the bosses when working my way up.

Sirius · Post by **Sirius** » Fri Dec 01, 2006 7:57 pm

I've had thoughts about doing so myself DFW, but kind of lack the time to make much progress. That and I'm tied up with the level right now... taking one thing at a time I guess...

Diedel - if they appear correctly in D2X-XL I would guess that would be all you need, and I suspect Polytron is fine for that (provided it doesn't screw up the centre of submodels - which govern how the robot joint positions rotate them).

Chaos Death Saurer · Post by **Chaos Death Saurer** » Fri Dec 01, 2006 8:28 pm

I'm working on the D1 L7 boss right now. I'll finish him off, then begin on the last D2 level. We can meet in the middle.

I assume the D1 bots will get the same treatment?

DarkFlameWolf · Post by **DarkFlameWolf** » Fri Dec 01, 2006 10:01 pm

They should get the same treatment, same goes for the Vertigo robots. All custom robots in custom missions campaigns, unfortunately are gonna be left out at the moment I'd imagine.

Diedel · Post by **Diedel** » Sat Dec 02, 2006 3:25 am

Yes, D1 bots get the same treatment. Btw, I have fixed that orange drone from D1 level 1, too.

You need to make sure that adjacent polygons really share vertices. The P.E.S.T. had many polygons that seemed to be connected but had disjunct vertices even where they should have shared them. I had to remove 3/4 of its hull, delete all but one of the overlapping vertices per corner, and rebuild it. Ack.

Make sure to remove unused vertices, too.

One hint about submodels: I frequently had it that Polytron started to complain about several submodels sharing vertices when deleting a vertex in a robot model with submodels. It is a good idea to move all submodels back to the main model before starting to work on it, and make them submodels again when you're finished.

I have also notices that some models have overlapping polygons, e.g. at the gun muzzles, where the muzzle is formed by a smaller polygon inside a bigger one. In such a case you have to remove the bigger one and connect the corners of the gun's front side with the corresponding corners of the muzzle polygon.

I can only recommend to make backup copies frequently, and make several sets (like pest1.pol, pest2.pol, pest3.pol) to save several stages of your work in case you screw up one.

Btw, CDS, you can also do the D1 robots, and I will do the D2 robots. Just let me know what you're gonna do. It might be a good idea to use a systematic approach though, and starting with a boss may not be the best warmup.

Sirius,

I don't know why Polytron should mess up the model center. I don't move any vertices, I only remove duplicate ones, and use the existing vertices to cleanly rebuild the model.

Sirius · Post by **Sirius** » Sat Dec 02, 2006 6:44 am

Well, if it doesn't touch the stuff in the meanwhile, it shouldn't. But from working on some robots in past, I know that it tends to screw up BSP data anyway (presumably the internal representation is either different or changed from the version in the file; it doesn't keep all the SORTNORM blocks at any rate), so who knows whether it automatically calculates submodel centres on save or just stores them from the file...

Bizarrely, for removing points working with the raw data wouldn't be much more difficult - all you'd have to do would be to strip out entries in the relevant DEFP_START block and change all the affected point indices for the affected polygons. Of course, that creates its own hassle since polymodels also contain quite a lot of internal pointers to be updated... and you also have to find the duplicate points in the first place.

Wish there were a better tool for doing that work though. HAXMEDIT/DOS doesn't let you edit very much \"to preserve data integrity\", which really means the author didn't have the time to make it recalculate all that stuff so elected to protect users from screwing things up instead.

At some point in the past I wrote a tool that can handle those kind of low-level changes, but sadly it still has all the pointer crap in it because I hadn't figured out what they were for at that point. (Edit: I also spotted odd bugs that I had no explanation for, but that's another story - and I could possibly fix them by now.)

Chaos Death Saurer · Post by **Chaos Death Saurer** » Sat Dec 02, 2006 9:00 am

Deidel:

Let's deal with the D2 bots first, then go for D1. someone else will have to do the Vertigo bots, as I don't have Vertigo.

We could do every other system for D2. You take systems 1, 3, 5, and I'll do 2, 4, and 6. Then we can get to D1. We can divide those robots up, 10 for each of us.

DarkFlameWolf · Post by **DarkFlameWolf** » Sat Dec 02, 2006 10:58 am

I can send you Vertigo for this process.

Diedel · Post by **Diedel** » Sat Dec 02, 2006 11:04 am

CDS,

I'd rather have you work down and me up, alright? I am done with the Thief and will do level 2 next (unless you have started on it already - if so, let me know).

Another model debugging tip:

If I cannot easily detect a flaw in a model causing shadowing to fail, I save it to some file named 'test.pol' and remove all parts of the model that are alright. I also remove all unused vertices resulting from that process. Next I count all faces and vertices of the reduced model: That way I find out very fast if I am having overlapping vertices and/or faces.

I have built a very simple cubic level with a single light, containing just one robot plus the player. That way I can easily check the model I am just working on. Btw, the fastest way to launch this level in D2X-XL is to load the level in DLE-XP and press Ctrl+F5. That way you can also easily change the level and re-check.

You can turn off robot movement by adding -nobotai to d2x.ini to make that easier.

To add test.pol to the hxm file, I right click on the model's icon in hxmedit32, chose 'replace' and load test.pol. That way you can also reload your original fixed model into the hxm file.

As I told you before, I move all submodels back to the main model before starting to work on the model. Before I do that, I sketch a little schematic on a piece of paper and number the model parts according to their submodel number to help me put the submodels back in the correct sequence once I am done.

----------

Btw, did none of you notice the shadow artifacts on the models themselves? Self shadowing didn't work ... just fixed that.

I am a volume shadow coding expert now ... I have seen some nice articles about soft shadows ... but ... no.

Chaos Death Saurer · Post by **Chaos Death Saurer** » Sat Dec 02, 2006 1:51 pm

I'll get started on level 24 when I get home, then.

And I learned very quickly working on the D1 L7 boss to compress everything into one model. Makes life easier.

And personally, unless the submodel is supposed to move, I'll leave them like that. Unless I'm not supposed to (I've noticed no difference while testing the L7 boss).

DarkFlameWolf: Send Vertigo to unicronthedestroyer@gmail.com. Thanks.

Diedel · Post by **Diedel** » Sat Dec 02, 2006 7:52 pm

CDS,

if a model has submodels, these will fly away separately when the object explodes.

Chaos Death Saurer · Post by **Chaos Death Saurer** » Sat Dec 02, 2006 8:50 pm

Ah. So that's what they're for.

Diedel · Post by **Diedel** » Sun Dec 03, 2006 3:30 am

They're mainly there to make a model have moving parts, flying away on destruction is just another feature.

DarkFlameWolf · Post by **DarkFlameWolf** » Sun Dec 03, 2006 9:03 am

Even I knew that!

But seriously, you should have Vertigo now CDS.

Also, the shadows now look much better on the models that haev been fixed, but for some reason, they still look weird in execution. Maybe because they aren't really grounded to a surface like normal FPS's do with their enemies. So it has to account for shadows on all sides. So maybe that's what I'm not used to.
Now, however, Descent 2 XL now seems to have an annoying tendancy to crash out upon quitting a level/game. Not sure what causes it.

Chaos Death Saurer · Post by **Chaos Death Saurer** » Sun Dec 03, 2006 9:18 am

I knew submodels made it move, but that's about it.

And yes, I got the Vertigo bots. Thanks.

Diedel · Post by **Diedel** » Sun Dec 03, 2006 2:49 pm

Wolf,

I don't understand what you mean with 'execution', why shadows don't look good. One thing is that you have light sources from all sides in D2, which can make shadows appear and disappear suddenly. You can try to use 2 or 3 light sources for shadowing if your hardware is fast enough.

CDS,

another important hint on model fixing. Adjacent faces need to share edges. Edges are shared if the share the same vertices. That means that something like:

x-----x-----x (edge with 3 vertices)
x------------x (edge with 2 vertices)

(Actually the edges should overlap completely).

although being parallel edges is separate edges. That means you need to remove intermediate vertices on face's edges. This can require removing and rebuilding several faces of the model.

Btw, I have fixed all bot models up to and including D2 level 4 except the boss.

Chaos Death Saurer · Post by **Chaos Death Saurer** » Sun Dec 03, 2006 3:47 pm

I'm *mostly* done with the Green Spider, I just need to fiddle with the claws. The Green Spawn thing is done, though. If D2X and DLE would cooperate, I'd get to see how they look.

Diedel · Post by **Diedel** » Sun Dec 03, 2006 4:51 pm

They do cooperate. Start D2X-XL from DLE-XP by pressing Ctrl+F5. You need to configure the D2X-XL folder first though (-> Preferences).

Please send your shadow proof models as .pol files to karx11erx at hotmail dot com. You can e.g. save them from Polytron.

Diedel · Post by **Diedel** » Mon Dec 04, 2006 6:54 pm

Dang. Shadows go through walls, and I have no idea how to fix that.

Shadowfury333 · Post by **Shadowfury333** » Mon Dec 04, 2006 7:49 pm

Diedel wrote:Dang. Shadows go through walls, and I have no idea how to fix that.

I think the only way to fix that is to have the walls cast shadows as well, thus the respective lights will not shine through the walls. The shadows will still pass through, but no one will notice (Doom 3 did this).

Diedel · Post by **Diedel** » Tue Dec 05, 2006 4:33 am

Actually that means rendering light from every light source ...

DarkFlameWolf · Post by **DarkFlameWolf** » Tue Dec 05, 2006 6:33 am

woohoo ^_^ Coding hell for sure Diedel. But you're always up for a challenge eh?

Shadowfury333 · Post by **Shadowfury333** » Tue Dec 05, 2006 7:44 am

DarkFlameWolf wrote:woohoo ^_^ Coding hell for sure Diedel. But you're always up for a challenge eh?

Isn't it just a simple recursion?

Diedel · Post by **Diedel** » Tue Dec 05, 2006 9:13 am

For many light sources that will mean it gets too slow.

The problem with the walls casting shadows is that these shadows will darken areas they cannot really reach as well.

Chaos Death Saurer · Post by **Chaos Death Saurer** » Tue Dec 05, 2006 6:35 pm

1. Apparently I'm not done with any of the models. The Green Spider's legs keep casting wonky shadows, so I may have to make them immobile to keep them closed (the individual submodels are closed. I think I have to move them.)

2. I was trying to get some footage of Lunar Outpost Reloaded for the video, and the shadows are going all over the place. It looks ugly.

I've got it set at three light sources. I'll try turning it down, but it won't be as pretty.

3. I've noticed that, with robot shadowing on, they do cast shadows, and where the shadows go wonky is usually where the problem is. That should help pinpoint the errors.

4. Lasers shouldn't be casting shadows.

Diedel · Post by **Diedel** » Tue Dec 05, 2006 6:56 pm

4. is a bug report (fixed though in next version).

If I find it hard to detect a model flaw, I start to remove all parts of the model except the one (or the first one) causing problems to pin the flaw down. I save that stripped down model in some test.pol. If I find something, I fix it and also fix it in the full model, until everything works. It is also easier to count faces and vertices for small model parts - I found quite a few duplicate faces and vertices that way. Btw, you can see excess vertices very well in one of the isometric views of Polytron.

Btw, I am working on preventing shadows from going through walls, but the solution is a big performance killer.