My reverse engineering tool

Show us what you're working on, (preferably with screenshots).
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

My reverse engineering tool

Post by TheGoodDoktor »

Hi

Over the last few weeks I've been working on a tool to reverse engineer Speccy games.
I used an open source emulator/debugger as a starting point and added extra features to help me reverse engineer games.
I built this for myself and I don't imagine many other people are interested in reverse engineering old games. I discovered Skoolkit a few months ago and ultimately I want this tool to generate output for it in some form.
It's in it's early stages at the moment and I'll release it (and the source) once it is in a good state - if anyone is interested of course.
If this tickles anyone's interest I can give you an version as I'd appreciate some feedback.

Image
User avatar
R-Tape
Site Admin
Posts: 6353
Joined: Thu Nov 09, 2017 11:46 am

Re: My reverse engineering tool

Post by R-Tape »

Ta for sharing, and a good example. A few of us have been unsuccessfully snorkeling through Firelord recently trying to find the bell chime sound.

What extra features does it have? This might be fanciful, but if a feature could somehow make it easier to find beeper routines that would be very cool.
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

OK, so here's a brief list of features:

Disassembler with label generation & ability to name them. Ability to annotate each line with comments.
Graphics viewer, view the whole address range as a bitmap with various formatting parameters (character map sizes, addressing modes)
Logging of memory accesses. For each memory location a list of every instruction address that either reads or writes to it.
Jump/call logging. For each jump location (a label) all source locations are recorded.
Logging of IO access. For each port all read & write instruction locations are logged - this is where you'd find your beeper code.
Double click on a screen pixel in the emulator and it'll take you to the instruction that wrote it.
Support for cheats.
Custom game content viewers (by modifying the source code of the tool atm.)

The tool analyses the game as it is being played so you need to try and hit all the code paths. RZX support would help this greatly - hopefully I can put this in some day.

They're are probably a load more features not mentioned, there's lots of stuff to ease code navigation. Generally speaking I've tried to reverse engineer a game and added features to help make that task easier in the process (if you see what I mean).
Ralf
Rick Dangerous
Posts: 2279
Joined: Mon Nov 13, 2017 11:59 am
Location: Poland

Re: My reverse engineering tool

Post by Ralf »

I would be interested.
Don't forget to write some documentation/tutorials ;)
User avatar
RMartins
Manic Miner
Posts: 776
Joined: Thu Nov 16, 2017 3:26 pm

Re: My reverse engineering tool

Post by RMartins »

Seems interesting ...
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

OK, So I've got a preview build that I can share with anyone who wants to try it.
DM me if you're interested.
User avatar
Lethargeek
Manic Miner
Posts: 734
Joined: Wed Dec 11, 2019 6:47 am

Re: My reverse engineering tool

Post by Lethargeek »

Very interesting to me. I did a lot of hacking of game code myself (even if only the graphics part) for my ULAX project with rather primitive assorted means (Spin for disasm listing, Xpeccy to find the sprites and trace code, and my own basic logging system). Would like to see if it would be much easier with everything integrated.
User avatar
PROSM
Manic Miner
Posts: 472
Joined: Fri Nov 17, 2017 7:18 pm
Location: Sunderland, England
Contact:

Re: My reverse engineering tool

Post by PROSM »

Now this looks really interesting. The emulator debuggers are okay for your own code, but are a pain when picking apart others' software. Having a whole suite of tools all in one program is very exciting. The memory access logging feature will become especially handy.
All software to-date
Working on something, as always.
User avatar
Lethargeek
Manic Miner
Posts: 734
Joined: Wed Dec 11, 2019 6:47 am

Re: My reverse engineering tool

Post by Lethargeek »

Tried it, my initial observations:

- Support for the sna format is a must;
- Snapshots to be put into a separate dedicated folder instead of creating a mess in the root folder;
- "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
- F5 key (i mean pressed on the keyboard, not clicked) doesn't work as Break, only as Continue;
- Graphics View arrangement is very inconvenient and uncomfortable. The slider and the view move in opposite directions and for some reason it can't go above #BFFF. Also rows of fixed height measured in characters is very bad idea for graphics view. Spectrum sprites are often of arbitrary pixel height, so at the very least the view should be arranged column by column of selected byte width. Even better if it will have the ability to reshuffle byte columns inside a sprite column (and then separately for even/odd pixel rows) and to filter out sprite descriptors breaking this view. Or even a simple scripting to describe sprite formats.

(didn't check other features yet)
Alone Coder
Manic Miner
Posts: 401
Joined: Fri Jan 03, 2020 10:00 am

Re: My reverse engineering tool

Post by Alone Coder »

Do you plan to release the decompiled sources?
User avatar
stupidget
Dynamite Dan
Posts: 1611
Joined: Wed Jan 24, 2018 2:09 pm
Location: Sunny Wolverhampton

Re: My reverse engineering tool

Post by stupidget »

R-Tape wrote: Sat Dec 28, 2019 8:46 pm Ta for sharing, and a good example. A few of us have been unsuccessfully snorkeling through Firelord recently trying to find the bell chime sound.
Ah, that amazing metallic sound is one of the many, many 'mind blown' events of my Speccy years.
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: My reverse engineering tool

Post by djnzx48 »

Lethargeek wrote: Fri Jan 03, 2020 2:39 pm - "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
Were you attempting to use 48K snapshots, or 128K snapshots (which don't appear to be supported)?
User avatar
PROSM
Manic Miner
Posts: 472
Joined: Fri Nov 17, 2017 7:18 pm
Location: Sunderland, England
Contact:

Re: My reverse engineering tool

Post by PROSM »

Lethargeek wrote: Fri Jan 03, 2020 2:39 pm - "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
I've been in contact with the creator about this issue, as I experienced it as well, and he replied that the tool does not yet support uncompressed Z80 snapshots. You need to save your snapshots in an emulator that has support for compression.
All software to-date
Working on something, as always.
User avatar
Lethargeek
Manic Miner
Posts: 734
Joined: Wed Dec 11, 2019 6:47 am

Re: My reverse engineering tool

Post by Lethargeek »

as i don't have many ready z80 snapshots, just created a 48k one in Spin at it worked; 128k one didn't

anyway, why not starting with sna support first? it is simpler and more suitable for any hacking
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

Thanks for the feedback. It was very useful and has definately given me some areas to focus on.
I've put some answers in-line.
Lethargeek wrote: Fri Jan 03, 2020 2:39 pm Tried it, my initial observations:

- Support for the sna format is a must;
I will put this in at some point. It didn't seem a high priority as most of the games I wanted to reverse engineer were available in .z80 format

- Snapshots to be put into a separate dedicated folder instead of creating a mess in the root folder;
Yes, I'm going to do this as it annoys me too!

- "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!

So what this is supposed to do is load the .z80 and internally set up the logic for reverse engineering the game. You can then save out the data by either choosing save or closing the application.
If as you say it is doing nothing then that could be due to a missing feature in the .z80 loader that someone else discovered the other day. This issue has been fixed and I'll release a new version shortly.

- F5 key (i mean pressed on the keyboard, not clicked) doesn't work as Break, only as Continue;
I'll have a look at this, could be a keyboard focus issue.

- Graphics View arrangement is very inconvenient and uncomfortable. The slider and the view move in opposite directions and for some reason it can't go above #BFFF. Also rows of fixed height measured in characters is very bad idea for graphics view. Spectrum sprites are often of arbitrary pixel height, so at the very least the view should be arranged column by column of selected byte width. Even better if it will have the ability to reshuffle byte columns inside a sprite column (and then separately for even/odd pixel rows) and to filter out sprite descriptors breaking this view. Or even a simple scripting to describe sprite formats.

Yes this is still very much WIP. The more games I try to reverse engineer, the more issues I find with it. Initially it was used to find the graphics.
The view goes only goes to #bfff because that's the address range minus how much can be displayed, It'll probably be better if I run off to blank rather than clamp the address range controls - I'll have a look.
The slider is a pain, I'll have another look at that.
I'll change the Y height to be in pixel rows rather than characters (or have a fine control as a lots of games use character heights.
I have been thinking about putting some scripting in but I want to look at a few more games before I formulate a plan. It would be good to know some games that it has difficulty viewing.


(didn't check other features yet)
User avatar
Lethargeek
Manic Miner
Posts: 734
Joined: Wed Dec 11, 2019 6:47 am

Re: My reverse engineering tool

Post by Lethargeek »

Even if you clamp the address range, the last one should have been #С000, not #BFFF. But yeah, better to blank the overflow or wrap around.

As how to arrange the view, first look at "sprite scanner" in Xpeccy (one column of selectable width) or repaint tool in EmuZWin (several such columns as fit in the window). Also it's useful to have possible width of column >32 bytes, as some games might use backbuffers wider than ZX screen.

As for the sprite formats, these are the examples i've seen:

- sprite made on the fly in the buffer out of 8x8 tiles
- sprite and mask separately, row by row
- sprite and mask side by side, row by row
- sprite and mask bytes interleaved in a row
- sprite and mask bytes reshuffled - (rick dangerous)
- sprite and mask are interleaved nibbles (!) - (opera soft games)
- interleaved sprite rows (the next row of specific sprite is its previous row+256)
- compressed mask of lower resolution than its sprite (but same byte width) - (capitan Trueno)

This might be complicated further with sprite descriptors put between the sprite pixel data shifting the beginning of the new sprite. Furthermore, bytes in the each row usually all go left to right but sometimes it is different for odd/even rows. Eg Commando sprites (3 bytes wide) have no mask, but even row bytes go left to right, odd row bytes right to left. I'm yet to see this combined with other methods, but won't be surprised.
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

Here's an updated version with some improvements to the graphics viewer, games directory & z80 loader fix:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

Lethargeek wrote: Sat Jan 04, 2020 3:29 pm Even if you clamp the address range, the last one should have been #С000, not #BFFF. But yeah, better to blank the overflow or wrap around.

As how to arrange the view, first look at "sprite scanner" in Xpeccy (one column of selectable width) or repaint tool in EmuZWin (several such columns as fit in the window). Also it's useful to have possible width of column >32 bytes, as some games might use backbuffers wider than ZX screen.

As for the sprite formats, these are the examples i've seen:

- sprite made on the fly in the buffer out of 8x8 tiles
- sprite and mask separately, row by row
- sprite and mask side by side, row by row
- sprite and mask bytes interleaved in a row
- sprite and mask bytes reshuffled - (rick dangerous)
- sprite and mask are interleaved nibbles (!) - (opera soft games)
- interleaved sprite rows (the next row of specific sprite is its previous row+256)
- compressed mask of lower resolution than its sprite (but same byte width) - (capitan Trueno)

This might be complicated further with sprite descriptors put between the sprite pixel data shifting the beginning of the new sprite. Furthermore, bytes in the each row usually all go left to right but sometimes it is different for odd/even rows. Eg Commando sprites (3 bytes wide) have no mask, but even row bytes go left to right, odd row bytes right to left. I'm yet to see this combined with other methods, but won't be surprised.
Thanks for the info.
I got Xpeccy but couldn't find how to access the sprite scanner - there doesn't seem to be any instructions. I also tried EmuZWin, I think its approach of displaying the sprites in columns is much better than my approach so I think i'll adopt it.
I appreciate some game references for the sprite layout examples you gave me (I know you gave a couple) - It would be really useful, a lot of the games I've looked at don't use masks, they use the 'good old XOR method'.

Cheers,
Mark.
User avatar
Lethargeek
Manic Miner
Posts: 734
Joined: Wed Dec 11, 2019 6:47 am

Re: My reverse engineering tool

Post by Lethargeek »

in Xpeccy press Esc to open the debugger window and then look for the small spanner icon in the Disasm section, there will be a pull-down menu
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

TheGoodDoktor wrote: Sat Dec 28, 2019 8:30 pm I don't imagine many other people are interested in reverse engineering old games.
I'm interested.
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: My reverse engineering tool

Post by djnzx48 »

Is it possible to change the type of a text/code/data segment if it's been set incorrectly? After setting memory to the code type, it doesn't seem possible to change it back, and there's no undo feature.

Also, are breakpoints able to be saved along with the project?
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

TheGoodDoktor wrote: Sat Jan 04, 2020 8:45 pm Here's an updated version with some improvements to the graphics viewer, games directory & z80 loader fix:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0
Seems, there are some dependecies.
Win 7:

Image

On another machine with Win 10 program is running normally.

P.S.
tried to get a bunch of DLLs, put these into program folder
XINPUT1_4.dll
api-ms-win-eventing-classicprovider-l1-1-0.dll
api-ms-win-core-sysinfo-l1-2-1.dll
api-ms-win-core-quirks-l1-1-0.dll
api-ms-win-core-libraryloader-l1-2-0.dll
api-ms-win-core-errorhandling-l1-1-1.dll
api-ms-win-core-synch-l1-2-0.dll
api-ms-win-core-processthreads-l1-1-2.dll
api-ms-win-core-io-l1-1-1.dll
api-ms-win-core-com-l1-1-1.dll
api-ms-win-core-file-l1-2-1.dll
api-ms-win-core-heap-l1-2-0.dll
api-ms-win-core-rtlsupport-l1-2-0.dll

but stuck with

Image
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

djnzx48 wrote: Fri Jan 10, 2020 6:15 am Is it possible to change the type of a text/code/data segment if it's been set incorrectly? After setting memory to the code type, it doesn't seem possible to change it back, and there's no undo feature.

Also, are breakpoints able to be saved along with the project?
Not sure it's in the last published version but you can use the 'C', 'D' & 'T' keys after selecting the line.
I'll be publishing a new version soon.
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: My reverse engineering tool

Post by druellan »

I'm not well versed on the Spectrum internals, but I find this kind of tools very valuable, specially visual inspectors that allows everyone to spot curious things.
Suggestion: a memory heatmap mode similar to the ones found on the Spud emulator:

Format 1:
Image

Fromat 2:
Image

Format 3:
Image

Format 4:
Image

It looks pretty nice in motion, you can easily spot the buffers and see them work realtime, but the emulator has a bug that misplaces the red and green overprints
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

druellan wrote: Thu Jan 16, 2020 9:29 pm Suggestion: a memory heatmap mode similar to the ones found on the Spud emulator:
How you achieved to run Spud?
It is always crying about missing ROM file, no matter what combination of ZX model/ROM I choose in options...

Image


P.S.
Ahhh, finally got it running.
Switched to default configuration.
Dont know why, but just extracting from archive and run doesnt't work. :lol:
Post Reply