My reverse engineering tool

Show us what you're working on, (preferably with screenshots).
User avatar
stupidget
Dynamite Dan
Posts: 1611
Joined: Wed Jan 24, 2018 2:09 pm
Location: Sunny Wolverhampton

Re: My reverse engineering tool

Post by stupidget »

R-Tape wrote: Sat Dec 28, 2019 8:46 pm Ta for sharing, and a good example. A few of us have been unsuccessfully snorkeling through Firelord recently trying to find the bell chime sound.
Ah, that amazing metallic sound is one of the many, many 'mind blown' events of my Speccy years.
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: My reverse engineering tool

Post by djnzx48 »

Lethargeek wrote: Fri Jan 03, 2020 2:39 pm - "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
Were you attempting to use 48K snapshots, or 128K snapshots (which don't appear to be supported)?
User avatar
PROSM
Manic Miner
Posts: 473
Joined: Fri Nov 17, 2017 7:18 pm
Location: Sunderland, England
Contact:

Re: My reverse engineering tool

Post by PROSM »

Lethargeek wrote: Fri Jan 03, 2020 2:39 pm - "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
I've been in contact with the creator about this issue, as I experienced it as well, and he replied that the tool does not yet support uncompressed Z80 snapshots. You need to save your snapshots in an emulator that has support for compression.
All software to-date
Working on something, as always.
User avatar
Lethargeek
Manic Miner
Posts: 735
Joined: Wed Dec 11, 2019 6:47 am

Re: My reverse engineering tool

Post by Lethargeek »

as i don't have many ready z80 snapshots, just created a 48k one in Spin at it worked; 128k one didn't

anyway, why not starting with sna support first? it is simpler and more suitable for any hacking
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

Thanks for the feedback. It was very useful and has definately given me some areas to focus on.
I've put some answers in-line.
Lethargeek wrote: Fri Jan 03, 2020 2:39 pm Tried it, my initial observations:

- Support for the sna format is a must;
I will put this in at some point. It didn't seem a high priority as most of the games I wanted to reverse engineer were available in .z80 format

- Snapshots to be put into a separate dedicated folder instead of creating a mess in the root folder;
Yes, I'm going to do this as it annoys me too!

- "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!

So what this is supposed to do is load the .z80 and internally set up the logic for reverse engineering the game. You can then save out the data by either choosing save or closing the application.
If as you say it is doing nothing then that could be due to a missing feature in the .z80 loader that someone else discovered the other day. This issue has been fixed and I'll release a new version shortly.

- F5 key (i mean pressed on the keyboard, not clicked) doesn't work as Break, only as Continue;
I'll have a look at this, could be a keyboard focus issue.

- Graphics View arrangement is very inconvenient and uncomfortable. The slider and the view move in opposite directions and for some reason it can't go above #BFFF. Also rows of fixed height measured in characters is very bad idea for graphics view. Spectrum sprites are often of arbitrary pixel height, so at the very least the view should be arranged column by column of selected byte width. Even better if it will have the ability to reshuffle byte columns inside a sprite column (and then separately for even/odd pixel rows) and to filter out sprite descriptors breaking this view. Or even a simple scripting to describe sprite formats.

Yes this is still very much WIP. The more games I try to reverse engineer, the more issues I find with it. Initially it was used to find the graphics.
The view goes only goes to #bfff because that's the address range minus how much can be displayed, It'll probably be better if I run off to blank rather than clamp the address range controls - I'll have a look.
The slider is a pain, I'll have another look at that.
I'll change the Y height to be in pixel rows rather than characters (or have a fine control as a lots of games use character heights.
I have been thinking about putting some scripting in but I want to look at a few more games before I formulate a plan. It would be good to know some games that it has difficulty viewing.


(didn't check other features yet)
User avatar
Lethargeek
Manic Miner
Posts: 735
Joined: Wed Dec 11, 2019 6:47 am

Re: My reverse engineering tool

Post by Lethargeek »

Even if you clamp the address range, the last one should have been #С000, not #BFFF. But yeah, better to blank the overflow or wrap around.

As how to arrange the view, first look at "sprite scanner" in Xpeccy (one column of selectable width) or repaint tool in EmuZWin (several such columns as fit in the window). Also it's useful to have possible width of column >32 bytes, as some games might use backbuffers wider than ZX screen.

As for the sprite formats, these are the examples i've seen:

- sprite made on the fly in the buffer out of 8x8 tiles
- sprite and mask separately, row by row
- sprite and mask side by side, row by row
- sprite and mask bytes interleaved in a row
- sprite and mask bytes reshuffled - (rick dangerous)
- sprite and mask are interleaved nibbles (!) - (opera soft games)
- interleaved sprite rows (the next row of specific sprite is its previous row+256)
- compressed mask of lower resolution than its sprite (but same byte width) - (capitan Trueno)

This might be complicated further with sprite descriptors put between the sprite pixel data shifting the beginning of the new sprite. Furthermore, bytes in the each row usually all go left to right but sometimes it is different for odd/even rows. Eg Commando sprites (3 bytes wide) have no mask, but even row bytes go left to right, odd row bytes right to left. I'm yet to see this combined with other methods, but won't be surprised.
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

Here's an updated version with some improvements to the graphics viewer, games directory & z80 loader fix:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

Lethargeek wrote: Sat Jan 04, 2020 3:29 pm Even if you clamp the address range, the last one should have been #С000, not #BFFF. But yeah, better to blank the overflow or wrap around.

As how to arrange the view, first look at "sprite scanner" in Xpeccy (one column of selectable width) or repaint tool in EmuZWin (several such columns as fit in the window). Also it's useful to have possible width of column >32 bytes, as some games might use backbuffers wider than ZX screen.

As for the sprite formats, these are the examples i've seen:

- sprite made on the fly in the buffer out of 8x8 tiles
- sprite and mask separately, row by row
- sprite and mask side by side, row by row
- sprite and mask bytes interleaved in a row
- sprite and mask bytes reshuffled - (rick dangerous)
- sprite and mask are interleaved nibbles (!) - (opera soft games)
- interleaved sprite rows (the next row of specific sprite is its previous row+256)
- compressed mask of lower resolution than its sprite (but same byte width) - (capitan Trueno)

This might be complicated further with sprite descriptors put between the sprite pixel data shifting the beginning of the new sprite. Furthermore, bytes in the each row usually all go left to right but sometimes it is different for odd/even rows. Eg Commando sprites (3 bytes wide) have no mask, but even row bytes go left to right, odd row bytes right to left. I'm yet to see this combined with other methods, but won't be surprised.
Thanks for the info.
I got Xpeccy but couldn't find how to access the sprite scanner - there doesn't seem to be any instructions. I also tried EmuZWin, I think its approach of displaying the sprites in columns is much better than my approach so I think i'll adopt it.
I appreciate some game references for the sprite layout examples you gave me (I know you gave a couple) - It would be really useful, a lot of the games I've looked at don't use masks, they use the 'good old XOR method'.

Cheers,
Mark.
User avatar
Lethargeek
Manic Miner
Posts: 735
Joined: Wed Dec 11, 2019 6:47 am

Re: My reverse engineering tool

Post by Lethargeek »

in Xpeccy press Esc to open the debugger window and then look for the small spanner icon in the Disasm section, there will be a pull-down menu
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

TheGoodDoktor wrote: Sat Dec 28, 2019 8:30 pm I don't imagine many other people are interested in reverse engineering old games.
I'm interested.
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: My reverse engineering tool

Post by djnzx48 »

Is it possible to change the type of a text/code/data segment if it's been set incorrectly? After setting memory to the code type, it doesn't seem possible to change it back, and there's no undo feature.

Also, are breakpoints able to be saved along with the project?
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

TheGoodDoktor wrote: Sat Jan 04, 2020 8:45 pm Here's an updated version with some improvements to the graphics viewer, games directory & z80 loader fix:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0
Seems, there are some dependecies.
Win 7:

Image

On another machine with Win 10 program is running normally.

P.S.
tried to get a bunch of DLLs, put these into program folder
XINPUT1_4.dll
api-ms-win-eventing-classicprovider-l1-1-0.dll
api-ms-win-core-sysinfo-l1-2-1.dll
api-ms-win-core-quirks-l1-1-0.dll
api-ms-win-core-libraryloader-l1-2-0.dll
api-ms-win-core-errorhandling-l1-1-1.dll
api-ms-win-core-synch-l1-2-0.dll
api-ms-win-core-processthreads-l1-1-2.dll
api-ms-win-core-io-l1-1-1.dll
api-ms-win-core-com-l1-1-1.dll
api-ms-win-core-file-l1-2-1.dll
api-ms-win-core-heap-l1-2-0.dll
api-ms-win-core-rtlsupport-l1-2-0.dll

but stuck with

Image
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

djnzx48 wrote: Fri Jan 10, 2020 6:15 am Is it possible to change the type of a text/code/data segment if it's been set incorrectly? After setting memory to the code type, it doesn't seem possible to change it back, and there's no undo feature.

Also, are breakpoints able to be saved along with the project?
Not sure it's in the last published version but you can use the 'C', 'D' & 'T' keys after selecting the line.
I'll be publishing a new version soon.
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: My reverse engineering tool

Post by druellan »

I'm not well versed on the Spectrum internals, but I find this kind of tools very valuable, specially visual inspectors that allows everyone to spot curious things.
Suggestion: a memory heatmap mode similar to the ones found on the Spud emulator:

Format 1:
Image

Fromat 2:
Image

Format 3:
Image

Format 4:
Image

It looks pretty nice in motion, you can easily spot the buffers and see them work realtime, but the emulator has a bug that misplaces the red and green overprints
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

druellan wrote: Thu Jan 16, 2020 9:29 pm Suggestion: a memory heatmap mode similar to the ones found on the Spud emulator:
How you achieved to run Spud?
It is always crying about missing ROM file, no matter what combination of ZX model/ROM I choose in options...

Image


P.S.
Ahhh, finally got it running.
Switched to default configuration.
Dont know why, but just extracting from archive and run doesnt't work. :lol:
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: My reverse engineering tool

Post by druellan »

This is how it looks working and with the read/write colors fixed. Sorry the size:

Image
TheGoodDoktor
Drutt
Posts: 9
Joined: Fri Dec 27, 2019 8:53 pm

Re: My reverse engineering tool

Post by TheGoodDoktor »

Thanks for all the recent feedback!
I've made several improvements to the graphics viewer:

Column based display
Heat map colourisation
Ability to select location & display info below

I've also improved the memory analysis, it was misses accesses before, it should get everything now.
There's some basic assembler output but it's experimental atm.
I was hoping to add .SNA support but ran out of time.
Any more feedback/suggestions will be gratefully received!

Here's the Dropbox link:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

TheGoodDoktor wrote: Sun Jan 19, 2020 4:52 pm Any more feedback/suggestions will be gratefully received!
Maybe switching to older xinput can solve Win 7 compatibility problem?
https://www.gamedev.net/forums/topic/69 ... l-missing/
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: My reverse engineering tool

Post by druellan »

Valley of Rains use attributes to hide unused sprites to speed up the code:

Image

Seems that the "screen" viewmode has no read/write indicators or perhaps I´m missing something?
User avatar
SkoolKid
Manic Miner
Posts: 403
Joined: Wed Nov 15, 2017 3:07 pm

Re: My reverse engineering tool

Post by SkoolKid »

TheGoodDoktor wrote: Sat Dec 28, 2019 8:30 pm I discovered Skoolkit a few months ago and ultimately I want this tool to generate output for it in some form
Just wanted to chime in and say I fully support this idea. :)

A few years ago someone else (apologies, don't remember who, and perhaps it would be impolite to name him, anyway) had the idea of making his Spectrum reverse engineering tool spit out SkoolKit control files, but in the end he gave up because control files are too complex (or something like that).

Now, yes, SkoolKit control files can be very complex, but they can also be very, very simple. So my advice, if you care to take it, would be to start off by generating the simplest possible control files, and then gradually work towards generating more complex ones.

Also, by the way, today is SkoolKit's 10th birthday!

https://skoolkit.ca/posts/2020/01/skoolkit-is-10/
SkoolKit - disassemble a game today
Pyskool - a remake of Skool Daze and Back to Skool
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

SkoolKid wrote: Tue Jan 28, 2020 5:00 pm Also, by the way, today is SkoolKit's 10th birthday!
Congratulations!
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: My reverse engineering tool

Post by djnzx48 »

I noticed that the floating bus doesn't seem to be emulated, so Cobra (for example) freezes as soon the game starts. Are there any plans to add this in?
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

Is there any further progress on this very interesting tool?
User avatar
Bedazzle
Manic Miner
Posts: 303
Joined: Sun Mar 24, 2019 9:03 am

Re: My reverse engineering tool

Post by Bedazzle »

Is it abandoned project now? :(
It had so big potential.
mrcook
Drutt
Posts: 46
Joined: Tue Jun 09, 2020 7:31 pm

Re: My reverse engineering tool

Post by mrcook »

Bedazzle wrote: Thu Oct 01, 2020 3:26 pm It had so big potential.
I just found this project but haven't been able to try it because - as is typical in the world of retro - it's a Windows only app and there's no source code available :/

From the shared image and comments this really did look promising. Shame.

EDIT: if you're interested, this tool seems to built on top of Andre Weissflog's "chips" tookbox (https://github.com/floooh/chips) - you can see the speccy debugging UI here: https://floooh.github.io/tiny8bit/zx-ui.html?type=zx48k
Post Reply