Page 1 of 2

Re: My reverse engineering tool

Posted: Fri Jan 03, 2020 5:04 pm
by stupidget
R-Tape wrote: Sat Dec 28, 2019 8:46 pm Ta for sharing, and a good example. A few of us have been unsuccessfully snorkeling through Firelord recently trying to find the bell chime sound.
Ah, that amazing metallic sound is one of the many, many 'mind blown' events of my Speccy years.

Re: My reverse engineering tool

Posted: Fri Jan 03, 2020 9:44 pm
by djnzx48
Lethargeek wrote: Fri Jan 03, 2020 2:39 pm - "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
Were you attempting to use 48K snapshots, or 128K snapshots (which don't appear to be supported)?

Re: My reverse engineering tool

Posted: Fri Jan 03, 2020 9:57 pm
by PROSM
Lethargeek wrote: Fri Jan 03, 2020 2:39 pm - "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!
I've been in contact with the creator about this issue, as I experienced it as well, and he replied that the tool does not yet support uncompressed Z80 snapshots. You need to save your snapshots in an emulator that has support for compression.

Re: My reverse engineering tool

Posted: Fri Jan 03, 2020 10:06 pm
by Lethargeek
as i don't have many ready z80 snapshots, just created a 48k one in Spin at it worked; 128k one didn't

anyway, why not starting with sna support first? it is simpler and more suitable for any hacking

Re: My reverse engineering tool

Posted: Sat Jan 04, 2020 12:27 pm
by TheGoodDoktor
Thanks for the feedback. It was very useful and has definately given me some areas to focus on.
I've put some answers in-line.
Lethargeek wrote: Fri Jan 03, 2020 2:39 pm Tried it, my initial observations:

- Support for the sna format is a must;
I will put this in at some point. It didn't seem a high priority as most of the games I wanted to reverse engineer were available in .z80 format

- Snapshots to be put into a separate dedicated folder instead of creating a mess in the root folder;
Yes, I'm going to do this as it annoys me too!

- "New game from z80 file" does literally nothing for any new snapshots (checked and working in other emulators) added by me. Even after the restart. I was able to load only the snapshots included with the preview (even those without any other data in the subfolders) but nothing else!

So what this is supposed to do is load the .z80 and internally set up the logic for reverse engineering the game. You can then save out the data by either choosing save or closing the application.
If as you say it is doing nothing then that could be due to a missing feature in the .z80 loader that someone else discovered the other day. This issue has been fixed and I'll release a new version shortly.

- F5 key (i mean pressed on the keyboard, not clicked) doesn't work as Break, only as Continue;
I'll have a look at this, could be a keyboard focus issue.

- Graphics View arrangement is very inconvenient and uncomfortable. The slider and the view move in opposite directions and for some reason it can't go above #BFFF. Also rows of fixed height measured in characters is very bad idea for graphics view. Spectrum sprites are often of arbitrary pixel height, so at the very least the view should be arranged column by column of selected byte width. Even better if it will have the ability to reshuffle byte columns inside a sprite column (and then separately for even/odd pixel rows) and to filter out sprite descriptors breaking this view. Or even a simple scripting to describe sprite formats.

Yes this is still very much WIP. The more games I try to reverse engineer, the more issues I find with it. Initially it was used to find the graphics.
The view goes only goes to #bfff because that's the address range minus how much can be displayed, It'll probably be better if I run off to blank rather than clamp the address range controls - I'll have a look.
The slider is a pain, I'll have another look at that.
I'll change the Y height to be in pixel rows rather than characters (or have a fine control as a lots of games use character heights.
I have been thinking about putting some scripting in but I want to look at a few more games before I formulate a plan. It would be good to know some games that it has difficulty viewing.


(didn't check other features yet)

Re: My reverse engineering tool

Posted: Sat Jan 04, 2020 3:29 pm
by Lethargeek
Even if you clamp the address range, the last one should have been #С000, not #BFFF. But yeah, better to blank the overflow or wrap around.

As how to arrange the view, first look at "sprite scanner" in Xpeccy (one column of selectable width) or repaint tool in EmuZWin (several such columns as fit in the window). Also it's useful to have possible width of column >32 bytes, as some games might use backbuffers wider than ZX screen.

As for the sprite formats, these are the examples i've seen:

- sprite made on the fly in the buffer out of 8x8 tiles
- sprite and mask separately, row by row
- sprite and mask side by side, row by row
- sprite and mask bytes interleaved in a row
- sprite and mask bytes reshuffled - (rick dangerous)
- sprite and mask are interleaved nibbles (!) - (opera soft games)
- interleaved sprite rows (the next row of specific sprite is its previous row+256)
- compressed mask of lower resolution than its sprite (but same byte width) - (capitan Trueno)

This might be complicated further with sprite descriptors put between the sprite pixel data shifting the beginning of the new sprite. Furthermore, bytes in the each row usually all go left to right but sometimes it is different for odd/even rows. Eg Commando sprites (3 bytes wide) have no mask, but even row bytes go left to right, odd row bytes right to left. I'm yet to see this combined with other methods, but won't be surprised.

Re: My reverse engineering tool

Posted: Sat Jan 04, 2020 8:45 pm
by TheGoodDoktor
Here's an updated version with some improvements to the graphics viewer, games directory & z80 loader fix:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0

Re: My reverse engineering tool

Posted: Sat Jan 04, 2020 9:08 pm
by TheGoodDoktor
Lethargeek wrote: Sat Jan 04, 2020 3:29 pm Even if you clamp the address range, the last one should have been #С000, not #BFFF. But yeah, better to blank the overflow or wrap around.

As how to arrange the view, first look at "sprite scanner" in Xpeccy (one column of selectable width) or repaint tool in EmuZWin (several such columns as fit in the window). Also it's useful to have possible width of column >32 bytes, as some games might use backbuffers wider than ZX screen.

As for the sprite formats, these are the examples i've seen:

- sprite made on the fly in the buffer out of 8x8 tiles
- sprite and mask separately, row by row
- sprite and mask side by side, row by row
- sprite and mask bytes interleaved in a row
- sprite and mask bytes reshuffled - (rick dangerous)
- sprite and mask are interleaved nibbles (!) - (opera soft games)
- interleaved sprite rows (the next row of specific sprite is its previous row+256)
- compressed mask of lower resolution than its sprite (but same byte width) - (capitan Trueno)

This might be complicated further with sprite descriptors put between the sprite pixel data shifting the beginning of the new sprite. Furthermore, bytes in the each row usually all go left to right but sometimes it is different for odd/even rows. Eg Commando sprites (3 bytes wide) have no mask, but even row bytes go left to right, odd row bytes right to left. I'm yet to see this combined with other methods, but won't be surprised.
Thanks for the info.
I got Xpeccy but couldn't find how to access the sprite scanner - there doesn't seem to be any instructions. I also tried EmuZWin, I think its approach of displaying the sprites in columns is much better than my approach so I think i'll adopt it.
I appreciate some game references for the sprite layout examples you gave me (I know you gave a couple) - It would be really useful, a lot of the games I've looked at don't use masks, they use the 'good old XOR method'.

Cheers,
Mark.

Re: My reverse engineering tool

Posted: Sat Jan 04, 2020 9:20 pm
by Lethargeek
in Xpeccy press Esc to open the debugger window and then look for the small spanner icon in the Disasm section, there will be a pull-down menu

Re: My reverse engineering tool

Posted: Fri Jan 10, 2020 4:58 am
by Bedazzle
TheGoodDoktor wrote: Sat Dec 28, 2019 8:30 pm I don't imagine many other people are interested in reverse engineering old games.
I'm interested.

Re: My reverse engineering tool

Posted: Fri Jan 10, 2020 6:15 am
by djnzx48
Is it possible to change the type of a text/code/data segment if it's been set incorrectly? After setting memory to the code type, it doesn't seem possible to change it back, and there's no undo feature.

Also, are breakpoints able to be saved along with the project?

Re: My reverse engineering tool

Posted: Sun Jan 12, 2020 9:22 am
by Bedazzle
TheGoodDoktor wrote: Sat Jan 04, 2020 8:45 pm Here's an updated version with some improvements to the graphics viewer, games directory & z80 loader fix:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0
Seems, there are some dependecies.
Win 7:

Image

On another machine with Win 10 program is running normally.

P.S.
tried to get a bunch of DLLs, put these into program folder
XINPUT1_4.dll
api-ms-win-eventing-classicprovider-l1-1-0.dll
api-ms-win-core-sysinfo-l1-2-1.dll
api-ms-win-core-quirks-l1-1-0.dll
api-ms-win-core-libraryloader-l1-2-0.dll
api-ms-win-core-errorhandling-l1-1-1.dll
api-ms-win-core-synch-l1-2-0.dll
api-ms-win-core-processthreads-l1-1-2.dll
api-ms-win-core-io-l1-1-1.dll
api-ms-win-core-com-l1-1-1.dll
api-ms-win-core-file-l1-2-1.dll
api-ms-win-core-heap-l1-2-0.dll
api-ms-win-core-rtlsupport-l1-2-0.dll

but stuck with

Image

Re: My reverse engineering tool

Posted: Sun Jan 12, 2020 9:47 pm
by TheGoodDoktor
djnzx48 wrote: Fri Jan 10, 2020 6:15 am Is it possible to change the type of a text/code/data segment if it's been set incorrectly? After setting memory to the code type, it doesn't seem possible to change it back, and there's no undo feature.

Also, are breakpoints able to be saved along with the project?
Not sure it's in the last published version but you can use the 'C', 'D' & 'T' keys after selecting the line.
I'll be publishing a new version soon.

Re: My reverse engineering tool

Posted: Thu Jan 16, 2020 9:29 pm
by druellan
I'm not well versed on the Spectrum internals, but I find this kind of tools very valuable, specially visual inspectors that allows everyone to spot curious things.
Suggestion: a memory heatmap mode similar to the ones found on the Spud emulator:

Format 1:
Image

Fromat 2:
Image

Format 3:
Image

Format 4:
Image

It looks pretty nice in motion, you can easily spot the buffers and see them work realtime, but the emulator has a bug that misplaces the red and green overprints

Re: My reverse engineering tool

Posted: Thu Jan 16, 2020 9:42 pm
by Bedazzle
druellan wrote: Thu Jan 16, 2020 9:29 pm Suggestion: a memory heatmap mode similar to the ones found on the Spud emulator:
How you achieved to run Spud?
It is always crying about missing ROM file, no matter what combination of ZX model/ROM I choose in options...

Image


P.S.
Ahhh, finally got it running.
Switched to default configuration.
Dont know why, but just extracting from archive and run doesnt't work. :lol:

Re: My reverse engineering tool

Posted: Fri Jan 17, 2020 1:23 am
by druellan
This is how it looks working and with the read/write colors fixed. Sorry the size:

Image

Re: My reverse engineering tool

Posted: Sun Jan 19, 2020 4:52 pm
by TheGoodDoktor
Thanks for all the recent feedback!
I've made several improvements to the graphics viewer:

Column based display
Heat map colourisation
Ability to select location & display info below

I've also improved the memory analysis, it was misses accesses before, it should get everything now.
There's some basic assembler output but it's experimental atm.
I was hoping to add .SNA support but ran out of time.
Any more feedback/suggestions will be gratefully received!

Here's the Dropbox link:
https://www.dropbox.com/sh/74olu8a70l14 ... nZQfa?dl=0

Re: My reverse engineering tool

Posted: Sun Jan 19, 2020 6:04 pm
by Bedazzle
TheGoodDoktor wrote: Sun Jan 19, 2020 4:52 pm Any more feedback/suggestions will be gratefully received!
Maybe switching to older xinput can solve Win 7 compatibility problem?
https://www.gamedev.net/forums/topic/69 ... l-missing/

Re: My reverse engineering tool

Posted: Tue Jan 21, 2020 2:05 pm
by druellan
Valley of Rains use attributes to hide unused sprites to speed up the code:

Image

Seems that the "screen" viewmode has no read/write indicators or perhaps I´m missing something?

Re: My reverse engineering tool

Posted: Tue Jan 28, 2020 5:00 pm
by SkoolKid
TheGoodDoktor wrote: Sat Dec 28, 2019 8:30 pm I discovered Skoolkit a few months ago and ultimately I want this tool to generate output for it in some form
Just wanted to chime in and say I fully support this idea. :)

A few years ago someone else (apologies, don't remember who, and perhaps it would be impolite to name him, anyway) had the idea of making his Spectrum reverse engineering tool spit out SkoolKit control files, but in the end he gave up because control files are too complex (or something like that).

Now, yes, SkoolKit control files can be very complex, but they can also be very, very simple. So my advice, if you care to take it, would be to start off by generating the simplest possible control files, and then gradually work towards generating more complex ones.

Also, by the way, today is SkoolKit's 10th birthday!

https://skoolkit.ca/posts/2020/01/skoolkit-is-10/

Re: My reverse engineering tool

Posted: Tue Jan 28, 2020 7:52 pm
by Bedazzle
SkoolKid wrote: Tue Jan 28, 2020 5:00 pm Also, by the way, today is SkoolKit's 10th birthday!
Congratulations!

Re: My reverse engineering tool

Posted: Tue Jan 28, 2020 10:34 pm
by djnzx48
I noticed that the floating bus doesn't seem to be emulated, so Cobra (for example) freezes as soon the game starts. Are there any plans to add this in?

Re: My reverse engineering tool

Posted: Wed Jun 03, 2020 7:01 am
by Bedazzle
Is there any further progress on this very interesting tool?

Re: My reverse engineering tool

Posted: Thu Oct 01, 2020 3:26 pm
by Bedazzle
Is it abandoned project now? :(
It had so big potential.

Re: My reverse engineering tool

Posted: Sun Oct 04, 2020 9:04 am
by mrcook
Bedazzle wrote: Thu Oct 01, 2020 3:26 pm It had so big potential.
I just found this project but haven't been able to try it because - as is typical in the world of retro - it's a Windows only app and there's no source code available :/

From the shared image and comments this really did look promising. Shame.

EDIT: if you're interested, this tool seems to built on top of Andre Weissflog's "chips" tookbox (https://github.com/floooh/chips) - you can see the speccy debugging UI here: https://floooh.github.io/tiny8bit/zx-ui.html?type=zx48k