SkoolKit: ctl file or skool file?

[dfzx]

Try this (limited time only):

Oh that's rather nice...

[RMartins]

Wait, you guys are using text highlighters?

I was using Notepad++ with no highlighting or text colours whatsoever

Notepad++ does some enlighting if you select ASM as file type (Although I'm not using it) and it always shows color help show matching parenthesis at least.

[Morkin]

Notepad++ does some enlighting if you select ASM as file type (Although I'm not using it) and it always shows color help show matching parenthesis at least.

Yep for years I didn't use Notepad++ or Visual Studio code for .asm, my eyes were well and truly opened afterwards...

[SkoolKid]

In light of the foregoing conversation, I've updated the "Disassembly DIY" documentation for SkoolKit. In particular, I've recommended developing the disassembly via the control file, and included a clear warning against editing the skool file for any purpose other than adding, removing or updating annotations:

https://skoolkit.readthedocs.io/en/late ... isassembly

I've even gone so far as to rename that section from "Developing the skool file" to "Developing the disassembly".

OK, end of lesson.

[lexi]

Catching up on this now.. I would be tempted to advocate switching to editing the .skool file to add annotations as having the assembly in-line is worth additional formatting to me - i currently spend probably twice as much time switching from referencing the HTML, switching back to the .ctl file and making the edit, running the ctl2skool and skool2html scripts, then switching back to the regenerated HTML only to then realise that visually the comment would be more readable when combined or split with the next few lines. It would be worth it to me to write some hooks so that a commit to either the .ctl or .skool would regenerate the other, allowing the choice of editing of each to be relatively transparent to each developer - leaving the choice to edit either up to personal comfort.

I can get close by experimenting with the command-line options, but when i convert an existing game.ctl file to game.skool, then convert that game.skool to game.0.ctl, and then finally convert that back to game.0.skool - it works but it's missing some critical references like the lists of which routines are called by other routines.

Is there support for this pipeline, or am i misunderstanding something critical - i was thinking of the .skool file as containing a superset of the data contained in a .ctl?

[mrcook]

i currently spend probably twice as much time switching from referencing the HTML, switching back to the .ctl file and making the edit, running the ctl2skool and skool2html scripts

In this situation I would normally create a Makefile so that I can run multiple commands in a single go. Here's a basic example: https://github.com/mrcook/skRex/blob/master/Makefile

It would be worth it to me to write some hooks so that a commit to either the .ctl or .skool would regenerate the other, allowing the choice of editing of each to be relatively transparent to each developer - leaving the choice to edit either up to personal comfort.

I don't like this kind of commit automation. If nothing else, the person editing the skool files would need to rebase after each commit...at some point you'd get yourself into a real mess of a merge conflict. But ultimately you're adding a lot of complexity for what is really just a hobby project

I can get close by experimenting with the command-line options, but when i convert an existing game.ctl file to game.skool, then convert that game.skool to game.0.ctl, and then finally convert that back to game.0.skool - it works but it's missing some critical references like the lists of which routines are called by other routines.

You don't need intermediary files here, just "rm -f game.skool && sna2skool.py ...", last I looked there wasn't a skoolkit command flag for forcing the overwrite, by maybe that's changed since.

I don't really like all those "routines ... called by other routines". Don't they even get out of sync? I haven't checked. They can be nice in the final ASM output, but I don't like them cluttering the control file. It would be nice if they could be disabled....or can they already?

i was thinking of the .skool file as containing a superset of the data contained in a .ctl?

As far as I know they are basically identical - even if the comment formatting looks different.

Skoolkid originally said this:

...I get the impression that reverse engineers these days generally prefer working on the control file exclusively. What about you?

In my opinion the control file is the most important file in a disassembly as it allows reproducibility. It also let's you easily reformat code/data. This is particularly useful when you thought something was a data block but is actually code: you can update everything with a single character change!

[Guesser]

I don't really like all those "routines ... called by other routines". Don't they even get out of sync? I haven't checked. They can be nice in the final ASM output, but I don't like them cluttering the control file. It would be nice if they could be disabled....or can they already?

They're not in the control file (or shouldn't be), that's part of the generated output.

[SkoolKid]

Catching up on this now.. I would be tempted to advocate switching to editing the .skool file to add annotations as having the assembly in-line is worth additional formatting to me...

That's my preferred approach too. Work on the control file until code and data are identified and distinguished (more or less). Then add annotations to the skool file, which is convenient because the assembly language instructions are right there. Go back to the control file briefly on the odd occasion that some significant code/data reorganisation is required.

...when i convert an existing game.ctl file to game.skool, then convert that game.skool to game.0.ctl, and then finally convert that back to game.0.skool - it works but it's missing some critical references like the lists of which routines are called by other routines.

That can happen because those lists are comments automatically generated by sna2skool.py. They appear by default only if no alternative comment is defined in the control file. To make sure that sna2skool.py always generates them, you can set the ListRefs configuration property to 2, as @Guesser and @szeliga already have in their skoolkit.ini.

[SkoolKid]

One more thing:

It would be worth it to me to write some hooks so that a commit to either the .ctl or .skool would regenerate the other...

My advice would be to never commit the skool file, because it can always be generated from the control file. When you're finished for the day on the skool file (if that's how you roll), use skool2ctl.py to update the control file, and commit that. The control file is rich enough to be the single source of truth for the disassembly.*

*Assuming you're all working from the same game snapshot - which you can ensure by using a suitable t2s file. (Hint hint.)

[lexi]

In this situation I would normally create a Makefile so that I can run multiple commands in a single go. Here's a basic example: https://github.com/mrcook/skRex/blob/master/Makefile

a Makefile approach is generally cleaner to me too but i have successfully avoided windows for long enough to know not whether the experience is better than the 2000s/cygwin/mingw days - i would not wish that upon anyone.

I don't like this kind of commit automation. If nothing else, the person editing the skool files would need to rebase after each commit...at some point you'd get yourself into a real mess of a merge conflict. But ultimately you're adding a lot of complexity for what is really just a hobby project

Yup - i dislike clever mechanisms whirring unexpectedly under my feet too. I'm okay with a cross-platfirm automated check in simple cases though - either:

• neither is changed
• one has been changed, but not the other - add Warning to commit msg as heads-up, or block
• both have changed, but the hook tests ctl2skool.sh and skool2ctl on both changed files and the copies are identical - so we're good
• the local dev has edited both in different ways - after testing ctl2skool and skool2ctl - there are now 2 different skool and 2 different ctl files - block

And agreed that hobby projects need to be fun to avoid being pointless.

And further plus also - there is no emoji yet contrived that can quite contain the multinefarious chalk board scratching sensations generated autonomously by my own suggestion of git hooks - so why would i even contemplate doing that to myself?

It's less about git-hooks specifically and more generally that I've found when working with devs that often it can be those most seemingly innocuous barriers which prevent an individual contributing and collaborating as much as they potentially want to, or at all - or worse. It could be a build process imposition, a personality conflict, previous merge-based trauma, a technological or ideological principle which if rationally-invalidated and optimistically-ignored rises with phoenix-beating-reliability from hill to become the volcano mount the project dies upon.

Sometimes that it means i'm in a more janitorial "clean up" role - writing tools, scripts and glue and manual tasks that no-one wants to do, but when done allows everyone to concentrate most on the parts of the work they do enjoy - there's no good metric for intangible team spirits, but it's satisfying when you see it clearly regardless

You don't need intermediary files here, just "rm -f game.skool && sna2skool.py ...", last I looked there wasn't a skoolkit command flag for forcing the overwrite, by maybe that's changed since.

I don't really like all those "routines ... called by other routines". Don't they even get out of sync? I haven't checked. They can be nice in the final ASM output, but I don't like them cluttering the control file. It would be nice if they could be disabled....or can they already?

This is what i mean - I wouldn't want the auto-generated refs to appear in the .ctl file, but they seemingly disappear during this process:

Code: Select all

; take a clean copy of the .ctl file, and make a copy of the .skool it generates
;
git checkout -- game.ctl
sna2skool.py -c game.ctl game.z80 > game.skool
cp game.skool game.skool-orig

; use the generated .skool file (containing a superset of the .ctl data?) to regenerate a 2nd .ctl file, replacing the first
;
skool2ctl.py -bh game.skool > game.ctl

; and use that regenerated .ctl file to create a 2nd .skool file using the original process
;
sna2skool.py -c game.ctl game.z80 > game.skool

; abridged diff - i'm fine with it reformatting that data-block, but all of the entry point data seems to have been stripped
;   out of the 2nd .skool file
;
diff game.skool game.skool-orig
...snip
< ; .
< ; This entry point is used by the routine at #R$D7EE.
11134,11135c10772
< b$D908 DEFB $00,$00                         ; #HTML(#DOUBLEFONT)
<  $D90A DEFB $00,$00,$00,$00,$00,$00
---
> b$D908 DEFB $00,$00,$00,$00,$00,$00,$00,$00 ; #HTML(#DOUBLEFONT)
11335,11336d10971
< ; .
< ; Used by the routine at #R$FE69.
11343,11344d10977
< ; .
< ; Used by the routines at #R$BAD6, #R$BB57 and #R$DF4E.
11404,11406d11036
< ; .
< ; Used by the routines at #R$90F0, #R$99F1, #R$9C59, #R$A18A, #R$B375, #R$B3C9,
< ; #R$B8DD, #R$BABE, #R$D763 and #R$DF72.
11430,11431d11059
< ; Used by the routine at #R$DF0C.
< ;
snip...

My take-away from Skoolkid's latest response that I'm probably just mis-using the tools - i'll go read the docs again more thoroughly

My advice would be to never commit the skool file, because it can always be generated from the control file. When you're finished for the day on the skool file (if that's how you roll), use skool2ctl.py to update the control file, and commit that. The control file is rich enough to be the single source of truth for the disassembly.*

Thanks for clarifying - this would be my preferred approach too - i just had difficulty getting it to work as i originally expected

*Assuming you're all working from the same game snapshot - which you can ensure by using a suitable t2s file. (Hint hint.)

ahhhhh... i deeply appreciate all this support - thankyou!

[SkoolKid]

...I wouldn't want the auto-generated refs to appear in the .ctl file...

Avoiding auto-generated ref comments in a control file that's regularly regenerated from a skool file is a bit tricky. skool2ctl.py does not (and cannot) identify auto-generated comments, so it will include them in its output. Perhaps once in a while you could manually remove them from the control file in order to have sna2skool.py regenerate them. If you're sure that any block description comment that starts with 'Used by the routine(s) at #R' is in fact an auto-generated comment, then something like this with GNU sed (and perhaps other species of sed) should work:

Code: Select all

$ sed -i '/^D ..... Used by the routines\? at #R/d' game.ctl

...but they seemingly disappear during this process,,,

I'm not sure how that could happen in the specific example you've given, which means you might have found a bug. In order for an auto-generated comment to "go missing", I would think you'd have to add a 'D' comment to the original control file that acts as a replacement and therefore suppresses it (unless ListRefs=2).

[Bedazzle]

skool2ctl.py does not (and cannot) identify auto-generated comments

But auto+generated comments can have some pre-defined pattern, usually not typed by human (for example, some non-visible unicode characters)

[SkoolKid]

But auto+generated comments can have some pre-defined pattern, usually not typed by human (for example, some non-visible unicode characters)

Using invisible characters to identify an auto-generated comment feels repugnant to me, and still not especially foolproof. I'd rather just add the option of omitting block description comments that match a regular expression such as "Used by the routines? at #R.*". Would anyone find that useful?

[lexi]

I'm not sure how that could happen in the specific example you've given, which means you might have found a bug. In order for an auto-generated comment to "go missing", I would think you'd have to add a 'D' comment to the original control file that acts as a replacement and therefore suppresses it (unless ListRefs=2).

This is encouraging - it's entirely probable I messed up configuration settings. I like the idea of being able to edit and format individual sections/routines in the non-tracked .skool on my end, and convert/paste those back into the git-tracked .ctl so that no-one else has to change their workflow/face unnecessary barriers to accommodate my preference

Using invisible characters to identify an auto-generated comment feels repugnant to me, and still not especially foolproof. I'd rather just add the option of omitting block description comments that match a regular expression such as "Used by the routines? at #R.*". Would anyone find that useful?

I have thoughts, but not enough direct experience to form a strong opinion - i feel like the "Used by routines.." call graph layer stuff is mixing auto-generated metadata with its source content. In the HTML it's invaluable as highlighting links, but does it need to live in the .skool also? If auto-generated content needs to remain in the .skool file for practical or design reasons - then i can't think of a good reason I'd want them transferred back into the .ctl file where the auto-generated call-graph data is treated as an authoritative source.

If i'm misunderstanding it slightly less incorrectly - if we discovered a new code-block late into the disassembly which calls a routine which already has the "Used by routines.." data generated for it.. someone could keep going without noticing that the newly discovered code-block doesn't trigger the auto-generated content to be regenerated - so that new reference is visible but undocumented?

I feel like this may intrude into the territory of stuff i should be aware of while disassembling and not expect the toolkit author to hold my hand every step of the way for - the memory map warned of dragons, and i ventured forth regardless

[SkoolKid]

I have thoughts, but not enough direct experience to form a strong opinion - i feel like the "Used by routines.." call graph layer stuff is mixing auto-generated metadata with its source content. In the HTML it's invaluable as highlighting links, but does it need to live in the .skool also? If auto-generated content needs to remain in the .skool file for practical or design reasons - then i can't think of a good reason I'd want them transferred back into the .ctl file where the auto-generated call-graph data is treated as an authoritative source.

sna2skool.py has been auto-generating the 'Used by' comments since SkoolKit 1.0.6. The idea was that the comments would be useful in the early stages of a disassembly when the control file reigns supreme, and would eventually become a permanent fixture in the skool file in the later stages of a disassembly (when you have all the code nailed down). I think it makes sense for them to be in the skool file, because then neither skool2html.py nor skool2asm.py (nor any other skool2X.py utility you care to implement) has to worry about generating them.

If i'm misunderstanding it slightly less incorrectly - if we discovered a new code-block late into the disassembly which calls a routine which already has the "Used by routines.." data generated for it.. someone could keep going without noticing that the newly discovered code-block doesn't trigger the auto-generated content to be regenerated - so that new reference is visible but undocumented?

Yes, that could happen if you're not careful. Just, um, be careful, I suppose!

I feel like this may intrude into the territory of stuff i should be aware of while disassembling and not expect the toolkit author to hold my hand every step of the way for - the memory map warned of dragons, and i ventured forth regardless

Perhaps a compromise is to use macros to generate the lists of callers. Back in SkoolKit 5 and earlier there were the #EREFS and #REFS macros that served that purpose. They were removed in SkoolKit 6, but you can use #FOREACH to simulate their behaviour. This way you can be sure that the 'Used by' comments won't go stale as you flit between the skool and control files. It does require the up-front investment of adding the macros to the control file to begin with, though.

[Guesser]

Just open the skool and control files side by side in a split screen editor, and every time you've added your next block of comments to the control file, hit the function key that runs the command to re-generate your skool file and read the beautifully formatted output that appears