My reverse engineering tool
-
- Drutt
- Posts: 9
- Joined: Fri Dec 27, 2019 8:53 pm
My reverse engineering tool
Hi
Over the last few weeks I've been working on a tool to reverse engineer Speccy games.
I used an open source emulator/debugger as a starting point and added extra features to help me reverse engineer games.
I built this for myself and I don't imagine many other people are interested in reverse engineering old games. I discovered Skoolkit a few months ago and ultimately I want this tool to generate output for it in some form.
It's in it's early stages at the moment and I'll release it (and the source) once it is in a good state - if anyone is interested of course.
If this tickles anyone's interest I can give you an version as I'd appreciate some feedback.
Over the last few weeks I've been working on a tool to reverse engineer Speccy games.
I used an open source emulator/debugger as a starting point and added extra features to help me reverse engineer games.
I built this for myself and I don't imagine many other people are interested in reverse engineering old games. I discovered Skoolkit a few months ago and ultimately I want this tool to generate output for it in some form.
It's in it's early stages at the moment and I'll release it (and the source) once it is in a good state - if anyone is interested of course.
If this tickles anyone's interest I can give you an version as I'd appreciate some feedback.
Re: My reverse engineering tool
Ta for sharing, and a good example. A few of us have been unsuccessfully snorkeling through Firelord recently trying to find the bell chime sound.
What extra features does it have? This might be fanciful, but if a feature could somehow make it easier to find beeper routines that would be very cool.
What extra features does it have? This might be fanciful, but if a feature could somehow make it easier to find beeper routines that would be very cool.
-
- Drutt
- Posts: 9
- Joined: Fri Dec 27, 2019 8:53 pm
Re: My reverse engineering tool
OK, so here's a brief list of features:
Disassembler with label generation & ability to name them. Ability to annotate each line with comments.
Graphics viewer, view the whole address range as a bitmap with various formatting parameters (character map sizes, addressing modes)
Logging of memory accesses. For each memory location a list of every instruction address that either reads or writes to it.
Jump/call logging. For each jump location (a label) all source locations are recorded.
Logging of IO access. For each port all read & write instruction locations are logged - this is where you'd find your beeper code.
Double click on a screen pixel in the emulator and it'll take you to the instruction that wrote it.
Support for cheats.
Custom game content viewers (by modifying the source code of the tool atm.)
The tool analyses the game as it is being played so you need to try and hit all the code paths. RZX support would help this greatly - hopefully I can put this in some day.
They're are probably a load more features not mentioned, there's lots of stuff to ease code navigation. Generally speaking I've tried to reverse engineer a game and added features to help make that task easier in the process (if you see what I mean).
Disassembler with label generation & ability to name them. Ability to annotate each line with comments.
Graphics viewer, view the whole address range as a bitmap with various formatting parameters (character map sizes, addressing modes)
Logging of memory accesses. For each memory location a list of every instruction address that either reads or writes to it.
Jump/call logging. For each jump location (a label) all source locations are recorded.
Logging of IO access. For each port all read & write instruction locations are logged - this is where you'd find your beeper code.
Double click on a screen pixel in the emulator and it'll take you to the instruction that wrote it.
Support for cheats.
Custom game content viewers (by modifying the source code of the tool atm.)
The tool analyses the game as it is being played so you need to try and hit all the code paths. RZX support would help this greatly - hopefully I can put this in some day.
They're are probably a load more features not mentioned, there's lots of stuff to ease code navigation. Generally speaking I've tried to reverse engineer a game and added features to help make that task easier in the process (if you see what I mean).
Re: My reverse engineering tool
I would be interested.
Don't forget to write some documentation/tutorials
Don't forget to write some documentation/tutorials
Re: My reverse engineering tool
Seems interesting ...
-
- Drutt
- Posts: 9
- Joined: Fri Dec 27, 2019 8:53 pm
Re: My reverse engineering tool
OK, So I've got a preview build that I can share with anyone who wants to try it.
DM me if you're interested.
DM me if you're interested.
- Lethargeek
- Manic Miner
- Posts: 759
- Joined: Wed Dec 11, 2019 6:47 am
Re: My reverse engineering tool
Very interesting to me. I did a lot of hacking of game code myself (even if only the graphics part) for my ULAX project with rather primitive assorted means (Spin for disasm listing, Xpeccy to find the sprites and trace code, and my own basic logging system). Would like to see if it would be much easier with everything integrated.
- PROSM
- Manic Miner
- Posts: 480
- Joined: Fri Nov 17, 2017 7:18 pm
- Location: Sunderland, England
- Contact:
Re: My reverse engineering tool
Now this looks really interesting. The emulator debuggers are okay for your own code, but are a pain when picking apart others' software. Having a whole suite of tools all in one program is very exciting. The memory access logging feature will become especially handy.
All software to-date
Working on something, as always.
Working on something, as always.
- Lethargeek
- Manic Miner
- Posts: 759
- Joined: Wed Dec 11, 2019 6:47 am
Re: My reverse engineering tool
Tried it, my initial observations:
- Support for the sna format is a must;
- Snapshots to be put into a separate dedicated folder instead of creating a mess in the root folder;
- "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
- F5 key (i mean pressed on the keyboard, not clicked) doesn't work as Break, only as Continue;
- Graphics View arrangement is very inconvenient and uncomfortable. The slider and the view move in opposite directions and for some reason it can't go above #BFFF. Also rows of fixed height measured in characters is very bad idea for graphics view. Spectrum sprites are often of arbitrary pixel height, so at the very least the view should be arranged column by column of selected byte width. Even better if it will have the ability to reshuffle byte columns inside a sprite column (and then separately for even/odd pixel rows) and to filter out sprite descriptors breaking this view. Or even a simple scripting to describe sprite formats.
(didn't check other features yet)
- Support for the sna format is a must;
- Snapshots to be put into a separate dedicated folder instead of creating a mess in the root folder;
- "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
- F5 key (i mean pressed on the keyboard, not clicked) doesn't work as Break, only as Continue;
- Graphics View arrangement is very inconvenient and uncomfortable. The slider and the view move in opposite directions and for some reason it can't go above #BFFF. Also rows of fixed height measured in characters is very bad idea for graphics view. Spectrum sprites are often of arbitrary pixel height, so at the very least the view should be arranged column by column of selected byte width. Even better if it will have the ability to reshuffle byte columns inside a sprite column (and then separately for even/odd pixel rows) and to filter out sprite descriptors breaking this view. Or even a simple scripting to describe sprite formats.
(didn't check other features yet)
-
- Manic Miner
- Posts: 401
- Joined: Fri Jan 03, 2020 10:00 am
Re: My reverse engineering tool
Do you plan to release the decompiled sources?
Re: My reverse engineering tool
Were you attempting to use 48K snapshots, or 128K snapshots (which don't appear to be supported)?Lethargeek wrote: ↑Fri Jan 03, 2020 2:39 pm - "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
- PROSM
- Manic Miner
- Posts: 480
- Joined: Fri Nov 17, 2017 7:18 pm
- Location: Sunderland, England
- Contact:
Re: My reverse engineering tool
I've been in contact with the creator about this issue, as I experienced it as well, and he replied that the tool does not yet support uncompressed Z80 snapshots. You need to save your snapshots in an emulator that has support for compression.Lethargeek wrote: ↑Fri Jan 03, 2020 2:39 pm - "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
All software to-date
Working on something, as always.
Working on something, as always.
- Lethargeek
- Manic Miner
- Posts: 759
- Joined: Wed Dec 11, 2019 6:47 am
Re: My reverse engineering tool
as i don't have many ready z80 snapshots, just created a 48k one in Spin at it worked; 128k one didn't
anyway, why not starting with sna support first? it is simpler and more suitable for any hacking
anyway, why not starting with sna support first? it is simpler and more suitable for any hacking
-
- Drutt
- Posts: 9
- Joined: Fri Dec 27, 2019 8:53 pm
Re: My reverse engineering tool
Thanks for the feedback. It was very useful and has definately given me some areas to focus on.
I've put some answers in-line.
I've put some answers in-line.
Lethargeek wrote: ↑Fri Jan 03, 2020 2:39 pm Tried it, my initial observations:
- Support for the sna format is a must;
I will put this in at some point. It didn't seem a high priority as most of the games I wanted to reverse engineer were available in .z80 format
- Snapshots to be put into a separate dedicated folder instead of creating a mess in the root folder;
Yes, I'm going to do this as it annoys me too!
- "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
So what this is supposed to do is load the .z80 and internally set up the logic for reverse engineering the game. You can then save out the data by either choosing save or closing the application.
If as you say it is doing nothing then that could be due to a missing feature in the .z80 loader that someone else discovered the other day. This issue has been fixed and I'll release a new version shortly.
- F5 key (i mean pressed on the keyboard, not clicked) doesn't work as Break, only as Continue;
I'll have a look at this, could be a keyboard focus issue.
- Graphics View arrangement is very inconvenient and uncomfortable. The slider and the view move in opposite directions and for some reason it can't go above #BFFF. Also rows of fixed height measured in characters is very bad idea for graphics view. Spectrum sprites are often of arbitrary pixel height, so at the very least the view should be arranged column by column of selected byte width. Even better if it will have the ability to reshuffle byte columns inside a sprite column (and then separately for even/odd pixel rows) and to filter out sprite descriptors breaking this view. Or even a simple scripting to describe sprite formats.
Yes this is still very much WIP. The more games I try to reverse engineer, the more issues I find with it. Initially it was used to find the graphics.
The view goes only goes to #bfff because that's the address range minus how much can be displayed, It'll probably be better if I run off to blank rather than clamp the address range controls - I'll have a look.
The slider is a pain, I'll have another look at that.
I'll change the Y height to be in pixel rows rather than characters (or have a fine control as a lots of games use character heights.
I have been thinking about putting some scripting in but I want to look at a few more games before I formulate a plan. It would be good to know some games that it has difficulty viewing.
(didn't check other features yet)
- Lethargeek
- Manic Miner
- Posts: 759
- Joined: Wed Dec 11, 2019 6:47 am
Re: My reverse engineering tool
Even if you clamp the address range, the last one should have been #С000, not #BFFF. But yeah, better to blank the overflow or wrap around.
As how to arrange the view, first look at "sprite scanner" in Xpeccy (one column of selectable width) or repaint tool in EmuZWin (several such columns as fit in the window). Also it's useful to have possible width of column >32 bytes, as some games might use backbuffers wider than ZX screen.
As for the sprite formats, these are the examples i've seen:
- sprite made on the fly in the buffer out of 8x8 tiles
- sprite and mask separately, row by row
- sprite and mask side by side, row by row
- sprite and mask bytes interleaved in a row
- sprite and mask bytes reshuffled - (rick dangerous)
- sprite and mask are interleaved nibbles (!) - (opera soft games)
- interleaved sprite rows (the next row of specific sprite is its previous row+256)
- compressed mask of lower resolution than its sprite (but same byte width) - (capitan Trueno)
This might be complicated further with sprite descriptors put between the sprite pixel data shifting the beginning of the new sprite. Furthermore, bytes in the each row usually all go left to right but sometimes it is different for odd/even rows. Eg Commando sprites (3 bytes wide) have no mask, but even row bytes go left to right, odd row bytes right to left. I'm yet to see this combined with other methods, but won't be surprised.
As how to arrange the view, first look at "sprite scanner" in Xpeccy (one column of selectable width) or repaint tool in EmuZWin (several such columns as fit in the window). Also it's useful to have possible width of column >32 bytes, as some games might use backbuffers wider than ZX screen.
As for the sprite formats, these are the examples i've seen:
- sprite made on the fly in the buffer out of 8x8 tiles
- sprite and mask separately, row by row
- sprite and mask side by side, row by row
- sprite and mask bytes interleaved in a row
- sprite and mask bytes reshuffled - (rick dangerous)
- sprite and mask are interleaved nibbles (!) - (opera soft games)
- interleaved sprite rows (the next row of specific sprite is its previous row+256)
- compressed mask of lower resolution than its sprite (but same byte width) - (capitan Trueno)
This might be complicated further with sprite descriptors put between the sprite pixel data shifting the beginning of the new sprite. Furthermore, bytes in the each row usually all go left to right but sometimes it is different for odd/even rows. Eg Commando sprites (3 bytes wide) have no mask, but even row bytes go left to right, odd row bytes right to left. I'm yet to see this combined with other methods, but won't be surprised.
-
- Drutt
- Posts: 9
- Joined: Fri Dec 27, 2019 8:53 pm
Re: My reverse engineering tool
Here's an updated version with some improvements to the graphics viewer, games directory & z80 loader fix:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0
-
- Drutt
- Posts: 9
- Joined: Fri Dec 27, 2019 8:53 pm
Re: My reverse engineering tool
Thanks for the info.Lethargeek wrote: ↑Sat Jan 04, 2020 3:29 pm Even if you clamp the address range, the last one should have been #С000, not #BFFF. But yeah, better to blank the overflow or wrap around.
As how to arrange the view, first look at "sprite scanner" in Xpeccy (one column of selectable width) or repaint tool in EmuZWin (several such columns as fit in the window). Also it's useful to have possible width of column >32 bytes, as some games might use backbuffers wider than ZX screen.
As for the sprite formats, these are the examples i've seen:
- sprite made on the fly in the buffer out of 8x8 tiles
- sprite and mask separately, row by row
- sprite and mask side by side, row by row
- sprite and mask bytes interleaved in a row
- sprite and mask bytes reshuffled - (rick dangerous)
- sprite and mask are interleaved nibbles (!) - (opera soft games)
- interleaved sprite rows (the next row of specific sprite is its previous row+256)
- compressed mask of lower resolution than its sprite (but same byte width) - (capitan Trueno)
This might be complicated further with sprite descriptors put between the sprite pixel data shifting the beginning of the new sprite. Furthermore, bytes in the each row usually all go left to right but sometimes it is different for odd/even rows. Eg Commando sprites (3 bytes wide) have no mask, but even row bytes go left to right, odd row bytes right to left. I'm yet to see this combined with other methods, but won't be surprised.
I got Xpeccy but couldn't find how to access the sprite scanner - there doesn't seem to be any instructions. I also tried EmuZWin, I think its approach of displaying the sprites in columns is much better than my approach so I think i'll adopt it.
I appreciate some game references for the sprite layout examples you gave me (I know you gave a couple) - It would be really useful, a lot of the games I've looked at don't use masks, they use the 'good old XOR method'.
Cheers,
Mark.
- Lethargeek
- Manic Miner
- Posts: 759
- Joined: Wed Dec 11, 2019 6:47 am
Re: My reverse engineering tool
in Xpeccy press Esc to open the debugger window and then look for the small spanner icon in the Disasm section, there will be a pull-down menu
Re: My reverse engineering tool
I'm interested.TheGoodDoktor wrote: ↑Sat Dec 28, 2019 8:30 pm I don't imagine many other people are interested in reverse engineering old games.
Re: My reverse engineering tool
Is it possible to change the type of a text/code/data segment if it's been set incorrectly? After setting memory to the code type, it doesn't seem possible to change it back, and there's no undo feature.
Also, are breakpoints able to be saved along with the project?
Also, are breakpoints able to be saved along with the project?
Re: My reverse engineering tool
Seems, there are some dependecies.TheGoodDoktor wrote: ↑Sat Jan 04, 2020 8:45 pm Here's an updated version with some improvements to the graphics viewer, games directory & z80 loader fix:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0
Win 7:
On another machine with Win 10 program is running normally.
P.S.
tried to get a bunch of DLLs, put these into program folder
XINPUT1_4.dll
api-ms-win-eventing-classicprovider-l1-1-0.dll
api-ms-win-core-sysinfo-l1-2-1.dll
api-ms-win-core-quirks-l1-1-0.dll
api-ms-win-core-libraryloader-l1-2-0.dll
api-ms-win-core-errorhandling-l1-1-1.dll
api-ms-win-core-synch-l1-2-0.dll
api-ms-win-core-processthreads-l1-1-2.dll
api-ms-win-core-io-l1-1-1.dll
api-ms-win-core-com-l1-1-1.dll
api-ms-win-core-file-l1-2-1.dll
api-ms-win-core-heap-l1-2-0.dll
api-ms-win-core-rtlsupport-l1-2-0.dll
but stuck with
-
- Drutt
- Posts: 9
- Joined: Fri Dec 27, 2019 8:53 pm
Re: My reverse engineering tool
Not sure it's in the last published version but you can use the 'C', 'D' & 'T' keys after selecting the line.djnzx48 wrote: ↑Fri Jan 10, 2020 6:15 am Is it possible to change the type of a text/code/data segment if it's been set incorrectly? After setting memory to the code type, it doesn't seem possible to change it back, and there's no undo feature.
Also, are breakpoints able to be saved along with the project?
I'll be publishing a new version soon.
Re: My reverse engineering tool
I'm not well versed on the Spectrum internals, but I find this kind of tools very valuable, specially visual inspectors that allows everyone to spot curious things.
Suggestion: a memory heatmap mode similar to the ones found on the Spud emulator:
Format 1:
Fromat 2:
Format 3:
Format 4:
It looks pretty nice in motion, you can easily spot the buffers and see them work realtime, but the emulator has a bug that misplaces the red and green overprints
Suggestion: a memory heatmap mode similar to the ones found on the Spud emulator:
Format 1:
Fromat 2:
Format 3:
Format 4:
It looks pretty nice in motion, you can easily spot the buffers and see them work realtime, but the emulator has a bug that misplaces the red and green overprints
Re: My reverse engineering tool
How you achieved to run Spud?
It is always crying about missing ROM file, no matter what combination of ZX model/ROM I choose in options...
P.S.
Ahhh, finally got it running.
Switched to default configuration.
Dont know why, but just extracting from archive and run doesnt't work.