Fix for Linux Loki Checksum bug

[MoonSire]

Thank you so much!! Your work is very appreciated!!

[The Lion]

Ok, but still... it looks like a simple LD_PRELOAD job to me.

Job done btw.

[MoonSire]

@TheLion

woohoo! Thank you! So many talanted people and all willing to help! This really made my day! Thanks!

[The Lion]

You're welcome.

It was an easy job though, as I expected.

[Arcalyth]

Sorry for the bump, but can anyone mirror the fix? Munk's site 404s and I'd love to play something other than the campaign!

[Munk]

I'm ashamed it's 404, i have no clue how that happened (VEX changed servers twice since the release).

Searching the web I found the perl script
http://odf.sourceforge.net/downloads/descent3/d3fixd.pl
but you will also need the checksum database.

Things on my todo list (after finishing studies in a month) is a better version of that fix based on LD_PRELOAD hacks instead of IP mangling. It would not need any root privileges...

If you can wait a few weeks, I would need some help in testing.

[flip]

http://www.filehosting.org/file/details ... les.tar.gz

http://www.filehosting.org/file/details ... 1.6.tgz.gz

[Arcalyth]

Thanks flip. Now to get it working... having trouble getting IPTables::IPv4 to install properly >_<

Munk, I'd be happy to assist you in testing. Gmail: Arcalyth
Go ahead and send me some mail when you're ready.

[flip]

NP, over the last 6 years I think I downloaded everything Descent related I could find on the web

[Arcalyth]

I haven't been able to get d3fixd working because IPTables::IPv4 module refuses to install. Since I'm impatient, I figured I'd try to write a python version since the iptables module for python works. To do so, I need the original checksums file. I tried to use Storable to deserialize the one in the .tar.gz, but the new version of Storable can't read it and the old one that you linked on Page 2 won't compile either.

So, do you have the original non-serialized checksums?

[flip]

No, it doesn't look like it, just those 2.

[akula65]

If you are referring to this link ( http://munk.vex-server.de/d3fix/checksums ), there is one instance of it in the Wayback Machine ( http://archive.org/ ) in 2007.

[Jeff250]

I'm taking a stab at this bug from the other direction by hacking at the linux binary's x86/x87 machine code. The following python script can be applied to both the statically and the dynamically linked linux binaries to reconcile the "sqrt" bug:

Code: Select all

# obsolete, see post below

It looks like that when loading the *.dlv file, at some point D3 calculates the normal vectors for a series of surfaces, presumably walls and such, but I haven't dug deep enough yet to figure out what for sure. It then normalizes these vectors to unit length, which is when sqrt() is applied. The windows binary returns the result of this sqrt() in "long double", i.e., in an x87 80-bit precision register, whereas the linux binary first rounds the result to "float" by popping from an x87 register to a 32-bit precision stack location, then pushing back into the x87 register. The checksum is somehow a function of these unit-length normal vectors. The above patch removes the linux binary's rounding of the sqrt() result.

I've tested this patch on a number of popular levels. The results are below. The first Y/N is whether the checksums agreed before the above patch. The second Y/N is whether they agreed after the patch.

abend 2: N Y
ascent: N Y
burning indika 3: Y Y
halcyon: Y Y
kata 1.2: N Y
pyroglyphic: Y Y
skybox: Y Y
stadium: Y Y
subway dancer: N N

The good news is that this patch hasn't introduced any regressions in my testing. Moreover, it reconciles the checksums for a number of levels. Subway Dancer, however, still produces disparate checksums, and I suspect some others untested may as well.

This means that the "sqrt" bug wasn't the only bug.

Is there some reason why Subway Dancer might still be affected, i.e., does it use some feature that perhaps the other levels don't? I've already tried stripping its *.gam file and other customizations from its *.mn3 (neither of which actually appear to even be checksummed...), and it still fails. It may just be hitting some other rounding issue...

Note to programmers: don't use floating point arithmetic for anything where an '==' comparison might matter. Unless your compiler is C99 conforming (AFAIK Microsoft's isn't), where the compiler rounds floating point arithmetic is undefined.

Maybe something simpler than tracking down the other discrepancy(s) is to just jump around the checksum check all together. Would the community care if I did this? It could theoretically be used by linux users for cheating, but I think there are already a dozen easier ways of doing that. The checksum is still a nice sanity check to make sure you aren't accidentally using the wrong version of a level though...

[Munk]

Good work, very nice !

I know for sure that the selected player ship is included into the checksum. Subway dancer uses modified game content, which could also be included.

[Jeff250]

I've found where the checksums are kept in memory. Both keep it as a 32-bit int stored in statically allocated memory: Windows keeps it at 0x5a6648, whereas linux keeps it at 0x82c0484. Knowing this makes it a lot easier to debug this issue. I've also found where the checksums are checked, but I don't think we will need to disable that, as I've found another floating point rounding discrepancy that accounts for the mismatch I was still seeing.

This discrepancy was the opposite case: Windows rounded from 80-bit to 32-bit precision when linux didn't. The fix for this discrepancy resolves the mismatch with, e.g., Subway Dancer. The following script can be used to patch either the statically or the dynamically linked linux binaries, and it's also comprehensive, including the patch from my earlier post:

Code: Select all

#!/usr/bin/env python
# Patches either the statically or the dynamically linked D3 Loki executables
# to resolve multiplayer level checksum mismatches with the Windows version
# By Jeff250

PATCHES = [
    (
        [
            (0, '\x55\x89\xe5\x83\xec\x18\x8b\x45\x08\xd9\x00\xd9\x40\x04\xd9\xc9\xd8\xc8\xd9\xc9\xd8\xc8\xde\xc1\xd9\x40\x08\xd8\xc8\xde\xc1\xd9\xfa\xd9\x5d\xfc\xd9\x45\xfc\x89\xec\x5d\xc3'),
        ],
        [
            (33, '\x90\x90\x90\x90\x90\x90'),
        ]
    ),
    (
        [
            (0, '\x55\x89\xe5\x83\xec\x3c\x57\x56\x53\x8b\x75\x0c\xd9\xee\x31\xc9\x39\xf1\x7d\x76\x8b\x5d\x10\x8d\x41\x02\x99\xf7\xfe\x8b\x7d\x14\x0f\xbf\x04\x53\x8d\x04\x40\x8d\x04\x87\x50\x8b\x7d\x10\x8d\x59\x01\x89\xd8\x99\xf7\xfe\x0f\xbf\x04\x57\x8b\x55\x14\x8d\x04\x40\x8d\x04\x82\x50\x0f\xbf\x04\x4f\x8d\x04\x40\x8d\x04\x82\x50\x8d\x45\xf4\x50\xdb\x7d\xd0\xe8'),
            (4, '\x83\xc4\x10\xdb\x6d\xd0\xd8\xd1\xdf\xe0\x80\xe4\x05\x74\x18\xdd\xd8\x8b\x45\xf4\x8b\x55\x08\x89\x02\x8b\x45\xf8\x89\x42\x04\x8b\x45\xfc\x89\x42\x08\xeb\x02\xdd\xd9\x89\xd9\x39\xf1\x7c\x8a\xdc\x1d'),
            (4, '\xdf\xe0\x80\xe4\x45\x80\xfc\x01\x74\x06\xb0\x01\xeb\x04\x89\xf6\xb0\x00\x8d\x65\xb8\x5b\x5e\x5f\x89\xec\x5d\xc3'),
        ],
        [
            (83, '\xd9\x5d\xd0'),
            (8, '\xd9\x45\xd0'),
        ],
    ),
]

def patch(path):
    try:
        with open(path, 'rb') as f:
            buf = f.read()
    except IOError:
        print '[-] Cannot read %s' % path
        exit(1)
    print '[+] Read %s' % path
    patched = list(buf)
    for i, (before, after) in enumerate(PATCHES):
        first_skip, first_find = before[0]
        start = 0
        found = 0
        while True:
            start = buf.find(first_find, start + first_skip) - first_skip
            if start < 0:
                break
            offset = start + first_skip + len(first_find)
            for skip, find in before[1:]:
                offset += skip
                offset = buf.find(find, offset, offset + len(find))
                if offset < 0:
                    break
                offset += len(find)
            if offset >= 0:
                for skip, replace in after:
                    start += skip
                    patched[start:start + len(replace)] = replace
                    start += len(replace)
                print '[+] Offset #%d to patch found at %#x' % (i + 1, start)
                found += 1
            else:
                start += 1
        if found < 1:
            print '[-] Cannot find offset #%d to patch' % (i + 1)
            exit(1)
    backup_path = path + '.old'
    try:
        with open(backup_path, 'wb') as f:
            f.write(buf)
    except IOError:
        print '[-] Cannot save backup to %s' % backup_path
        exit(1)
    print '[+] Saved backup to %s' % backup_path
    try:
        with open(path, 'wb') as f:
            f.write(''.join(patched))
    except IOError:
        print '[-] Cannot write patches to %s' % path
        exit(1)
    print '[+] Wrote patch to %s' % path

if __name__ == '__main__':
    import sys
    if len(sys.argv) != 2:
        sys.stderr.write('Usage: %s EXECUTABLE\n' % sys.argv[0])
        exit(1)
    patch(sys.argv[1])

Note: This patch will fail if you've already patched using the script from my earlier post--restore the backup first!

This should now be ready for mass consumption. All levels that I've tested now produce consistent checksums. If anyone finds levels that mismatch despite using the above patch, please report them.

edit: Attached script for download below.
edit #2: Cleaned up python, but patch itself is unchanged.
edit #3: Fixed for statically linked binary.

[theAntiBob]

Been trying to patch descent3 and descent3.dynamic with d3-loki-checksum-patch.py.txt (renamed to not txt)
descent3.dynamic patches fine (or says it does) but doesn't launch when called, before or after being patched. just does this:

Code: Select all

: /usr/local/games/descent3/descent3$ ./descent3.dynamic 
./descent3.dynamic: error while loading shared libraries: libSDL-1.2.so.0: cannot open shared object file: No such file or directory
:/usr/local/games/descent3$ locate libSDL-1.2.so.0
/usr/lib/x86_64-linux-gnu/libSDL-1.2.so.0
/usr/lib/x86_64-linux-gnu/libSDL-1.2.so.0.11.4
:/usr/local/games/descent3$ sudo ~/Downloads/descent3/./d3-loki-checksum-patch.py descent3.dynamic 
[sudo] password for theantibob: 
[+] Read descent3.dynamic
[+] Offset #1 to patch found at 0x20e327
[+] Offset #2 to patch found at 0x134608
[+] Saved backup to descent3.dynamic.old
[+] Wrote patch to descent3.dynamic
:/usr/local/games/descent3$ ./descent3.dynamic
./descent3.dynamic: error while loading shared libraries: libSDL-1.2.so.0: cannot open shared object file: No such file or directory

do i need a symlink somewhere?
when attempting to patch descent3, this happens:

Code: Select all

$ sudo ./d3-loki-checksum-patch.py /usr/local/games/descent3/descent3
[+] Read /usr/local/games/descent3/descent3
[+] Offset #1 to patch found at 0x211a77
[-] Cannot find offset #2 to patch

any idea why this won't patch?
it looks like other people have had success with this.

i just wanna play!

btw, the loki port works flawlessly for me, with the exception of not being able to render in 1920x1080. 1600x1200 works well enough on this display so it's still MUCH better than through wine!

[Jeff250]

when attempting to patch descent3, this happens:

Code: Select all

$ sudo ./d3-loki-checksum-patch.py /usr/local/games/descent3/descent3
[+] Read /usr/local/games/descent3/descent3
[+] Offset #1 to patch found at 0x211a77
[-] Cannot find offset #2 to patch

Oops, I've fixed the script in my earlier post to again work with the statically linked binary.

descent3.dynamic patches fine (or says it does) but doesn't launch when called, before or after being patched. just does this:
[...]
do i need a symlink somewhere?

The dynamically linked binary didn't work before the patch either, right? In order to get the dynamically linked binary to work, you need to hunt down the dependencies for it. For instance, right now it's complaining about not having libsdl, so you'll need to get that. (On Ubuntu, "sudo apt-get install libsdl1.2debian:i386".)

[theAntiBob]

Thanks for the update! Both descent3 and descent3.dynamic were successfully patched.
Works great and I can FINALLY play d3 online... (not like there's really anyone out there to play) I do have a stash of win98 pentiumIII boxes that may get used for a lanparty, though.

The dynamically linked binary didn't work before the patch either, right? In order to get the dynamically linked binary to work, you need to hunt down the dependencies for it. For instance, right now it's complaining about not having libsdl, so you'll need to get that. (On Ubuntu, "sudo apt-get install libsdl1.2debian:i386".)

No, it wasn't working before... did:

Code: Select all

$sudo apt-get install libsdl1.2debian:i386 libsmpeg0:i386

but now it just gives me sig11. I'm not really too concerned.

Thanks for all your help and effort...

and thank you, SpiffyBruin for letting me be theAntiBob again.

[Jeff250]

but now it just gives me sig11. I'm not really too concerned.

descent3.dynamic tends to not play nice on newer systems with newer versions of the libraries it links against. The Outrage or Loki folks probably assumed something about an API that they weren't supposed to and that got changed at some point. But like you say, there's no reason to be concerned about it as long as the statically linked binary works. One caveat with the statically linked binary is that it has a version of libsdl built in that is hardcoded to use OSS, the old linux sound system, so you need to run it through the appropriate emulation layer for sound to work. For pulseaudio:

Code: Select all

padsp /usr/local/games/Descent3/descent3 -n # and rest of args

For straight up ALSA:

Code: Select all

aoss /usr/local/games/Descent3/descent3 -n # and rest of args

I suspect you already figured this out, but I'm just documenting it here for whoever else might read this thread someday.