Z88DK and Fuse, list level debugging

The place for codemasters or beginners to talk about programming any language for the Spectrum.
Post Reply
dfzx
Manic Miner
Posts: 673
Joined: Mon Nov 13, 2017 6:55 pm
Location: New Forest, UK
Contact:

Z88DK and Fuse, list level debugging

Post by dfzx »

I posted this to the Z88DK list which is probably where it belongs, but that's a low traffic environment and one or two people here might also find it interesting. So with apologies for the cross post:

Since I started with Z88DK it's become apparent that I've been a bit spoilt over the years with source level debugging. Now I haven't got it I realise how much I rely on it. When my Spectrum program doesn't work the options appear to be to either stare at the C code hoping for inspiration, or single step through the generated Z80 code hoping to be able to follow what's going on. I went looking for an alternative.

The map file produced when you give the -m flag to zcc produces a table of symbols and their addresses in the final executable. Adding the --list option along with the --c-code-in-asm option produce listing files containing the generated assembly language, commented with the original C code. Given that the map contains the addresses of all the functions, and the *.c.lis files contain the offset of each Z80 instruction from the the start of the list file, with a bit of munging it's possible to work out the absolute final address of any instruction in the compiled program.

I wrote a Perl script to do this munging: it takes the map and listing files, works out all the instruction addresses, then concatenates the listing files to generate a single listing file where each line is pre-pended with the instruction address. So it looks, for example, like this:

Code: Select all

...
0x9079 ++               l_gameloop_00129:
0x9079 ++               ;gameloop.c:164: if( in_key_pressed( IN_KEY_SCANCODE_SPACE ) ) {
0x9079 ++   21 7F 01            ld      hl,0x017f
0x907C ++   CD 00 00            call    _in_key_pressed_fastcall
0x907F ++   4D                  ld      c, l
0x9080 ++   7C                  ld      a, h
0x9081 ++   B1                  or      a,c
0x9082 ++   28 0A               jr      Z,l_gameloop_00104
0x9084 ++               ;gameloop.c:165: game_state->key_pressed = 1;
0x9084 ++   DD 6E FA            ld      l,(ix-6)
0x9087 ++   DD 66 FB            ld      h,(ix-5)
0x908A ++   36 01               ld      (hl),0x01
0x908C ++   18 10               jr      l_gameloop_00141
0x908E ++               l_gameloop_00104:
0x908E ++               ;gameloop.c:167: game_state->key_pressed = 0;
0x908E ++   DD 6E FA            ld      l,(ix-6)
0x9091 ++   DD 66 FB            ld      h,(ix-5)
0x9094 ++   36 00               ld      (hl),0x00
...
The "0xXXXX ++" tag on the left side is what my script adds to the single output file - the address of each instruction. I refer to this file as the "tagged source".

I then went to the Fuse emulator debugger source and added a Gtk text widget into the window. The Gtk text widget has the concept of "marks" which allow a point in the text to be identified by a name, a short text string. The widget seems happy to have thousands of these marks, so I wrote some code to load my tagged source file into it, using each line's address tag as a text widget mark. This allows the Gtk text widget containing my tagged source to be able to find and scroll the text to the instruction at any given address.

Final step: the Fuse debugger has a hook which allows the dialog to be updated as the emulator advances the Z80 program counter, so I hooked my Gtk text widget to that with a bit of code which takes the PC register, converts it to a string, finds that string as a mark in the text and scrolls to it. The effect is that as you click the "step" button in the Fuse debugger, the C-and-assembler mixed code in the text window follows the program counter.

Obviously you're still stepping though Z80 instructions, not C code, and you don't get the see the local variables unless you pay attention to the registers or the position in the stack frame they're in. But you do get the context of the C program. You can see the C code the Z80 is working though, and you can see all the symbols which are in that C code, including function names, statically allocated variables and compiler generated labels. It's not hard to see how the Z80 flags are controlling loops, etc.

I recorded a short desktop video, showing it working:

[media]https://youtu.be/TmpHy_x2Fko[/media]

I don't know how useful anyone else might find this, but for me, at my level of experience, I'm finding myself using it all the time.

Fuse is free software and I have a fork on SourceForge. I'm not sure where this is going yet, so can't really provide a static link to the fork. If you want a copy of the Perl script or the Fuse fork, by all means PM me.
Derek Fountain, author of the ZX Spectrum C Programmer's Getting Started Guide and various open source games, hardware and other projects, including an IF1 and ZX Microdrive emulator.
User avatar
Fred
Drutt
Posts: 38
Joined: Tue Feb 27, 2018 3:15 am

Re: Z88DK and Fuse, list level debugging

Post by Fred »

That looks really cool :D
User avatar
Stefan
Manic Miner
Posts: 792
Joined: Mon Nov 13, 2017 9:51 pm
Location: Belgium
Contact:

Re: Z88DK and Fuse, list level debugging

Post by Stefan »

Cool! As a quick win, some very basic colour coding just of c vs assembly lines could make it even easier to use.
User avatar
bob_fossil
Manic Miner
Posts: 654
Joined: Mon Nov 13, 2017 6:09 pm

Re: Z88DK and Fuse, list level debugging

Post by bob_fossil »

Cheers for doing this and explaining the map and list files.

I've been using a modified version of fbzx to debug my Z88DK projects. I added a basic debugger and the ability to break into the emulator from the source code by inlining in a 'do nothing' opcode, so I could look at the assembly at a specific point to see what I was doing wrong.

For Knockabout, I was banging my head trying to debug a function using border colours to show where I'd got to in it and dumping variables out to screen memory to read with the debugger as I didn't know the location of the variables. In the end I took a different approach and took the C code and turned it into a console app on my PC using the terminal to output basic graphics to the screen. With the aid of a proper IDE I found the problem in minutes.

I used a similar approach for debugging and developing the C end of Thoroughly Modern Willy, though I upgraded the graphics to use SDL on the PC (I created a surface the same resolution as the Spectrum screen). Through some #ifdefs I ended up with code I could compile for the PC or Spectrum. I still had to manually debug the assembly on the Spectrum but the ability to do watches and add breakpoints in the IDE made it a lot easier to try things out and fix problems.

Using your explanation I've added a symbol lookup to my custom fbzx so I can now disassemble or dump memory from a symbol in the map file as well as an absolute address. I've also started to add a very basic .lis parser so I can get the source code for a given address.
dfzx
Manic Miner
Posts: 673
Joined: Mon Nov 13, 2017 6:55 pm
Location: New Forest, UK
Contact:

Re: Z88DK and Fuse, list level debugging

Post by dfzx »

I can't help a great deal with your travails, Bob, but it was somehow comforting to know I'm not the only with these problems. :) You're clearly further forward than I am; I've not written any Spectrum program of use as yet, and certainly not a finished game.

I've used exactly the same trick with border colours, it being the only simple way to communicate with the user as to program location or variable value. Not exactly an advanced interface though. :lol:

I also thought about writing some sort of compatibility layer which would allow a Spectrum C program to be compiled for a PC environment, but given that I'm interested in the SP1 library I'd need to get a pretty accurate emulation of the Spectrum memory, screen and Z80 working before it would be useful. I thought about using something like QEmu, but decided that improving the Spectrum emulator tooling would probably be more productive. I'm still very much in the realm of making it up as I go along though. :)

I think there's a bit more legs in this tagged source idea yet. One thing that comes to mind is to genericise it: instead of my address tag on each line, put something there carrying a bit more information. A small JSON block comes to mind, which could carry various values relevant to the generating environment - address, syntax hinting as @Stefan suggested, an indicator of whether a line is C code in a comment, Z80 opcode, assembler directive like DEFB, etc. The presentation level (Fuse) could then read that information and use it to show, hide, highlight, etc., different lines and types as required. What I can't see at the moment is a way to get variable values visible in the emulator's debugger. It would require a way to get C variable types and sizes, plus the location the compiler has put them in the stack frame.

Before any of that stuff, I'd like to find a way to get symbols into the Fuse debugger. Or a decent memory map explorer. And probably quite a few other things. :)
Derek Fountain, author of the ZX Spectrum C Programmer's Getting Started Guide and various open source games, hardware and other projects, including an IF1 and ZX Microdrive emulator.
User avatar
bob_fossil
Manic Miner
Posts: 654
Joined: Mon Nov 13, 2017 6:09 pm

Re: Z88DK and Fuse, list level debugging

Post by bob_fossil »

I still need to finish off Knockabout and Thoroughly Modern Willy was only six screens so I'm hardly a paragon of finished games. :)

I took a higher level approach and abstacted the graphics so that I could concentrate on the game logic. I'm not using SP1 as it's a bit of a black box and seems a bit memory hungry. It would have been overkill for the games where I'm just displaying 8x8 or 16x16 tiles to the screen.

I'm not sure if you can get the Z88dk compiler to generate files that describe structures and their types. One approach would be to manually create a specification file that names a structure and the types of it's members (or even write a basic parser which hunts for 'struct' in your C files). You could then load that in alongside the .map and .lis files and then have a debugger command to interpret a memory address as a named type (a bit like the dt command in windbg)
User avatar
jorgegv
Microbot
Posts: 102
Joined: Mon Aug 09, 2021 4:50 pm

Re: Z88DK and Fuse, list level debugging

Post by jorgegv »

Hi @dfzx , just stomped on this post, and I was wondering if you could post the Perl script that you use to generate the special tagged file that your FUSE fork uses for list level debugging. I managed to compile your fork, and I think your modifications would be really useful for me.

TIA

P.S.: btw, your top toolbar is quite useful also...
dfzx
Manic Miner
Posts: 673
Joined: Mon Nov 13, 2017 6:55 pm
Location: New Forest, UK
Contact:

Re: Z88DK and Fuse, list level debugging

Post by dfzx »

Oh man, this is getting on for 5 years old! I have a vague recollection from last year, or maybe the year before, of something changing, possibly the *.lis file format. It all stopped working. I'm trying to recall whether I got it fixed...

Give me some time, I'll dig it out and see where it's at. :)
Derek Fountain, author of the ZX Spectrum C Programmer's Getting Started Guide and various open source games, hardware and other projects, including an IF1 and ZX Microdrive emulator.
User avatar
jorgegv
Microbot
Posts: 102
Joined: Mon Aug 09, 2021 4:50 pm

Re: Z88DK and Fuse, list level debugging

Post by jorgegv »

I surely can help with that. I'm quite proficient with Perl, so if you put it in some shared repo, we can work on it together. I think your use case was almost exactly as mine, including our frustrations, so I'm really interested in getting it working again if needed. I don't mind diving into your FUSE fork either.
dfzx
Manic Miner
Posts: 673
Joined: Mon Nov 13, 2017 6:55 pm
Location: New Forest, UK
Contact:

Re: Z88DK and Fuse, list level debugging

Post by dfzx »

OK, I've had a look at this. I implemented my original idea for LIS-file debugging when developing my game Wonky One Key. The Perl which generates the tagged source file (i.e. the file loaded into the special version of Fuse's debugger) is here. But that's old and only works with z88dk v1.99 as far as I recall, so unless you're stuck with an old version of z88dk you can ignore that.

I updated the script when I was developing my later game, The Virus. I think that version was good for z88dk v2.0 or maybe v2.1. I can't recall. But the script stopped working when z88dk (SDCC, actually) changed the LIS file format again. In order to finish that game I stuck with the older version of z88dk because getting that script to work again proved a pain.

Your query pushed me to get the script working again, so I grabbed the latest z88dk (from 2 days ago) and have updated the script to work with it. The latest version of the script is here, as part of the source tree for my virus game. I should try to find somewhere better for it. :)

I suggest you grab the whole tree for The Virus onto a Linux box and try to build it. You might need an extra package or two, I can help with that. The makefile runs the tagged source script, so once you have it built you should be able to open the debugger of my Fuse fork, drag and drop the tagged source file into it, and hopefully it'll all work.

If you need help let me know. :)
Derek Fountain, author of the ZX Spectrum C Programmer's Getting Started Guide and various open source games, hardware and other projects, including an IF1 and ZX Microdrive emulator.
User avatar
jorgegv
Microbot
Posts: 102
Joined: Mon Aug 09, 2021 4:50 pm

Re: Z88DK and Fuse, list level debugging

Post by jorgegv »

Thanks Derek.

I cloned The Virus and got it working without any issues (well, I had to link ctags -> etags), and also I used your tagged source file in the debugger! Indeed this will speed up my debugging sessions!

Thanks so much, and I'll keep you informed of any changes I make. I have spotted a couple of issues on the generated tagged source, I'll fix and report back.

Do you have any plans of storing your fuse fork in Github instead of SF? Git support on SF is mediocre at best, and I'd like to send some some PRs for Fuse, I have some additional ideas in mind based on your changes...

Thanks again, @dfzx

P.S. Do you mind if I include your script in my RAGE1 sources? It would be great to integrate it into the RAGE1 devel workflow...
dfzx
Manic Miner
Posts: 673
Joined: Mon Nov 13, 2017 6:55 pm
Location: New Forest, UK
Contact:

Re: Z88DK and Fuse, list level debugging

Post by dfzx »

jorgegv wrote: Tue Jan 10, 2023 11:24 pm Do you have any plans of storing your fuse fork in Github instead of SF?
Not specifically. I've always been keen to work on an emulator and I did a bit of work on Fuse, but there was little reciprocation of anything I did. No one was interested. I kind of lost interest myself. The problem with moving it to Github is that it breaks the link to the original project. Then again, with one Fuse release in the last 4 years, that being almost 2 years ago, the original project is of limited value anyway.
jorgegv wrote: Tue Jan 10, 2023 11:24 pm P.S. Do you mind if I include your script in my RAGE1 sources? It would be great to integrate it into the RAGE1 devel workflow...
No, go ahead. I marked my script GPL which matches the RAGE1 licence. I'd be interested in your improvements, and keeping the script in sync with SDCC's LIS file format (which seems to be a moving target).
Derek Fountain, author of the ZX Spectrum C Programmer's Getting Started Guide and various open source games, hardware and other projects, including an IF1 and ZX Microdrive emulator.
User avatar
jorgegv
Microbot
Posts: 102
Joined: Mon Aug 09, 2021 4:50 pm

Re: Z88DK and Fuse, list level debugging

Post by jorgegv »

dfzx wrote: Thu Jan 12, 2023 11:21 am No, go ahead. I marked my script GPL which matches the RAGE1 licence. I'd be interested in your improvements, and keeping the script in sync with SDCC's LIS file format (which seems to be a moving target).
Mmm just thinking.... Wouldn't it be good if your script was part of Z88DK instead? It seems the logical place...

I think I'll ping the masters in Z88DK forums when I have the fixed script, and offer them a PR...

Thanks again, I'll keep you posted.
dfzx
Manic Miner
Posts: 673
Joined: Mon Nov 13, 2017 6:55 pm
Location: New Forest, UK
Contact:

Re: Z88DK and Fuse, list level debugging

Post by dfzx »

jorgegv wrote: Thu Jan 12, 2023 11:53 am Mmm just thinking.... Wouldn't it be good if your script was part of Z88DK instead? It seems the logical place...
Hmmm, not really. z88dk is a C development environment for Z80 based targets. What we're talking about here is an application for getting a Spectrum emulator's debugger to follow a source file. There's a broad overlap, but it's a rather specific application of the compiler's output IMO.

Someone else was recently interested, and asked me (privately) for more information. That got me thinking about making the concept more generic. The idea of "list level debugging" genericises into a Spectrum emulator's debugger offering a text window which scrolls to a particular place in its text based on some form of input from the emulator. In my application it's the Z80's PC register - if the PC holds, say, 0x8000, the text window scrolls to location marked 0x8000. But in theory the debugger could offer the value of any registers, the state of memory, the state of the screen, etc., and have the information window immediately and automatically scroll to a relevant location based on that data. It could show text, or a graphic, or maybe something else. It seems like a useful idea but I'm not entirely sure what, if anything, to do with it. :)
Derek Fountain, author of the ZX Spectrum C Programmer's Getting Started Guide and various open source games, hardware and other projects, including an IF1 and ZX Microdrive emulator.
User avatar
jorgegv
Microbot
Posts: 102
Joined: Mon Aug 09, 2021 4:50 pm

Re: Z88DK and Fuse, list level debugging

Post by jorgegv »

jorgegv wrote: Thu Jan 12, 2023 11:53 am I think I'll ping the masters in Z88DK forums when I have the fixed script, and offer them a PR...
Oh...wait...You are one of the Z88DK masters! :D
User avatar
jorgegv
Microbot
Posts: 102
Joined: Mon Aug 09, 2021 4:50 pm

Re: Z88DK and Fuse, list level debugging

Post by jorgegv »

Hey @dfzx , back on this one:

I have already added some small features to your tagged source debugging window in Fuse:
  • Full width "current line" - yeah, I'm used to that, it hurted me to see a half-width bar :)
  • Add a --tagged-src <filename> command line option to specify the tagged source file directly when running Fuse (drag'n'drop is not efficient for me :) )
  • Add RPM spec files and an autoconf fix so that I can update Fuse comfortably in Fedora - I opened a ticket to the Fuse Fedora maintaner, and still 2 years (and counting!) to change 1 line in the spec file to migrate from GTK2 to GTK3 - enough! :D
I forked your repo in Sourceforge and I made a Pull Request to yours just in case you are interested:

https://sourceforge.net/u/dfzx/fuse-emu ... equests/1/

And the link for my Fuse fork:

https://sourceforge.net/u/jorgegv/fuse/ci/master/tree/

I'll start thinking about more enhancements and the possible evolution of this. I really was missing a trivial source debugger for Fuse.

Cheers
J.
dfzx
Manic Miner
Posts: 673
Joined: Mon Nov 13, 2017 6:55 pm
Location: New Forest, UK
Contact:

Re: Z88DK and Fuse, list level debugging

Post by dfzx »

jorgegv wrote: Mon Jan 16, 2023 7:02 pm I forked your repo in Sourceforge and I made a Pull Request to yours just in case you are interested:
Thanks for that Jorge, I'd no idea how to do SourceForge merges, it gave me something to think about. :)

I merged your tagged debug improvements, but left the Fedora bits behind. I don't know anything about Fedora. Comments are in your merge request tracker.

My fork is now up to date with upstream Fuse, with my (and now your) bits on top. You should update everything in sight!
Derek Fountain, author of the ZX Spectrum C Programmer's Getting Started Guide and various open source games, hardware and other projects, including an IF1 and ZX Microdrive emulator.
Post Reply