SkoolKit: ctl file or skool file?

[SkoolKid]

This poll is inspired by [mention]cmal[/mention], who asked the million dollar question that polarises the SkoolKit-using community[1]: control file or skool file?

Back when I released SkoolKit 1.0, I expected users (if there were any) to use a control file in the initial stages of developing a disassembly, and then mostly work on the skool file. But I get the impression that reverse engineers these days generally prefer working on the control file exclusively. What about you?

[1] Maybe.

[pobtastic]

I only use the control file, but probably only because I've been using Skoolkit for so long and that was just how it "was" (or at least, how I assumed it to be). I always start from scratch too, with a blank file ~ not for any particular reason, that's just how I like to do it. One discovery at a time!

[Morkin]

Never touched the .skool file, always just worked on the .ctl file.

Then occasionally fire a sna2skool and a skool2html and refresh the local html page to see if it looks as I expect...

[Guesser]

Yup, I only edit the ctl file. I treat it like the "source code" of the disassembly and build everything else from scripts.

[cmal]

Thanks for posting the poll, [mention]SkoolKid[/mention] . Interesting result. So I'm clearly doing something wrong. I voted ".skool" file because that's what I've been doing so far.

It leads to the question:
If you edit the control file, how do you insert annotations that are adjacent to the disassembly commands?

My control file looks like this:
c 24092
c 24113
b 24134
c 24162
b 24205
t 24212
b 24215
t 24238

Skool file looks like this. It has the disassembly so I can add annotations:
; Used by the routines at #R24251, #R24276, #R24328 and #R24472.
c24113 LD HL,(24090) ; Restore A and Flags register
24116 PUSH HL ;
24117 POP AF ;
24118 LD IX,(24088) ; Restore IX
24122 LD HL,(24086) ; Restore HL
24125 LD DE,(24084) ; Restore DE
24129 LD BC,(24082) ; Restore BC
24133 RET ;

(anyone like to try guess what game this is?? )



Perhaps I have to run this step once, is that right?
"use skool2ctl.py to convert a disassembly back into a control file (with annotations retained)"
...then going forward, update the control file.

[SkoolKid]

Thanks for posting the poll, @SkoolKid . Interesting result. So I'm clearly doing something wrong. I voted ".skool" file because that's what I've been doing so far.

For what it's worth, I don't think there's anything wrong with editing and adding annotations to the skool file. That's what I've done for the majority of the disassemblies I've worked on. I like it because the code is right there in the text file, which makes it easy to annotate specific instructions as you go through a routine. If you're comfortable with skool file syntax, working on the skool file is a good option.

I suppose one advantage of working on the control file, though, is that control file syntax is a bit simpler: you don't have to worry about where your curly braces, semicolons and comment continuation lines are going, or how to format an entry header correctly.

It leads to the question:
If you edit the control file, how do you insert annotations that are adjacent to the disassembly commands?

For that you need to use sub-block syntax, which means the capital letter counterparts ('B', 'C', 'T' etc.) of the block directives ('b', 'c', 't' etc.).

And if you're wondering how sub-block syntax works, one way to figure that out is to follow your own advice:

Perhaps I have to run this step once, is that right?
"use skool2ctl.py to convert a disassembly back into a control file (with annotations retained)"
...then going forward, update the control file.

Although I usually prefer to work on skool files, I will still use skool2ctl.py to generate a (temporary) control file if I want to split up or join entries, or reorganise a data block (e.g. change the number of bytes per line) - because that's much easier to do in a control file.

[Bedazzle]

Sorry for offtopic, but...
Is it possible to "redirect" output to several files? For example, I do not like one long and huge file (asm or skool)?

[SkoolKid]

Is it possible to "redirect" output to several files? For example, I do not like one long and huge file (asm or skool)?

No, skool2asm.py and sna2skool.py just write to standard output, and there's no support for splitting up the output and sending it to more than one file.

Multiple invocations of skool2asm.py and sna2skool.py with carefully chosen --start and --end options might be a workaround, though.

Also, sna2skool.py does support reading multiple control files, so if you prefer control files to skool files, that is one way to split up your sources.

And obviously, skool2html.py already writes output to multiple files, because it would make no sense not to.

[nzeemin]

I did some (incomplete) disassemblies using the Skoolkit, and at some point I wonder if we have some modern IDE support (like syntax highlighting) for Skoolkit file formats.

Found SkoolkitZ80 for Sublime Text 3 editor - see https://github.com/mrcook/SkoolkitZ80 - that's for .skool files.

But, I am a .ctl file guy, so I made attempt to create a syntax highlighting plugin for VS Code: https://github.com/nzeemin/skoolkit-ctl-syntax
To use it with VS Code, git clone the repo or download it, and put in separate folder at %USERPROFILE%\.vscode\extensions
Hope it will be useful for someone other than me

[luny]

I did some (incomplete) disassemblies using the Skoolkit, and at some point I wonder if we have some modern IDE support (like syntax highlighting) for Skoolkit file formats.

Found SkoolkitZ80 for Sublime Text 3 editor - see https://github.com/mrcook/SkoolkitZ80 - that's for .skool files.

But, I am a .ctl file guy, so I made attempt to create a syntax highlighting plugin for VS Code: https://github.com/nzeemin/skoolkit-ctl-syntax
To use it with VS Code, git clone the repo or download it, and put in separate folder at %USERPROFILE%\.vscode\extensions
Hope it will be useful for someone other than me

I'm a CTL guy too. I'm going to try your new plugin for myself tonight. Thanks.

[dfzx]

Hmmm, I might be missing something...

I started my first SkoolKit disassembly this week, following the getting started guide. I created a CTL file as per the Hungry Horace example, then another based on a FUSE profiling run, built the Skool from them, then never touched them again.

The documentation seems to imply that once you have your Skool file, that's where you work from going onward. I didn't understand the bit where it explained why you might want to go back to the CTL file, so I left that until I needed it. So far I haven't needed it. Now I'm wondering why.

[Guesser]

It's up to the individual!
I see the ctl file as the "source" for mine, and keep my changes in version control, ignoring .skool as a generated output file that's just a stepping stone to html or asm. Scripts build everything else for me

It could just be that I'm lazy and don't want to do a lot of typing and text formatting

[SkoolKid]

I started my first SkoolKit disassembly this week, following the getting started guide. I created a CTL file as per the Hungry Horace example, then another based on a FUSE profiling run, built the Skool from them, then never touched them again.

The documentation seems to imply that once you have your Skool file, that's where you work from going onward. I didn't understand the bit where it explained why you might want to go back to the CTL file, so I left that until I needed it. So far I haven't needed it. Now I'm wondering why.

You're not missing anything. That Disassembly DIY documentation was written a long time ago, back when I expected users to start with a control file and eventually move to editing the skool file. But it seems most people prefer to stick with the control file and never touch the skool file. Which is fine, and means I should probably update the documentation to reflect that preference.

[RMartins]

You're not missing anything. That Disassembly DIY documentation was written a long time ago, back when I expected users to start with a control file and eventually move to editing the skool file. But it seems most people prefer to stick with the control file and never touch the skool file. Which is fine, and means I should probably update the documentation to reflect that preference.

Or document both process workflows.

This way one can choose one or the other, without the documentation forcing one or the other.

Note that I'm assuming that they are 100% equivalent, which might not be true.
It also seems that if they are equivalent, there is duplicated functionality between the Ctl and Skool file.

[SkoolKid]

Or document both process workflows.

I plan to update the documentation for the next release of SkoolKit (8.7).

There's really not much difference between the workflows, though. If you stick with a control file, you need to run sna2skool.py before running skool2html.py (or any other skool2* command). Whereas if you work on the skool file directly, you don't need to do that (because the skool file's always already up to date).

Note that I'm assuming that they are 100% equivalent, which might not be true.
It also seems that if they are equivalent, there is duplicated functionality between the Ctl and Skool file.

The skool file is derived from the control file (and a snapshot), but I don't think that makes them equivalent in any meaningful sense. That would be a bit like saying an object file produced by a compiler is equivalent to the source code that was used to build it.

[dfzx]

There's really not much difference between the workflows, though.

I wanted to work on the Skool file, but the thing which caught me out was the difficulty in converting from one type of content to another. For example, when I had this as a product of the Fuse profiler and sna2ctl.py:

Code: Select all

24477 	DEFM "+-e3"

and I realised that was actually 4 data bytes I wanted as 8 bit integers (i.e. DEFBs) there wasn't an obvious way to convert them other than by hand. Worse, when I realised some data was actually code and I wanted it converted from DEFB/DEFM statements to Z80 mnemonics (i.e. disassembled) that wasn't easily doable either, and doing that by hand really isn't an option in any non-trivial case.

The only obvious way to make such conversions is to go from skool back to ctl using the skool2ctl.py utility, tweak the ctl file, then go back from ctl to skool again. But it's a fussy process which involves an intermediate ctl file which I wasn't sure whether I should be keeping. It also loses information from the skool file: mostly formatting of my comments IIRC, but there was enough lost that as I got towards the end of my project that I stopped doing it.

I'm not sure what the answer is. It seems many people just stick with working on the ctl file, but I never really got the hang of it.

[SkoolKid]

The only obvious way to make such conversions is to go from skool back to ctl using the skool2ctl.py utility, tweak the ctl file, then go back from ctl to skool again. But it's a fussy process which involves an intermediate ctl file which I wasn't sure whether I should be keeping. It also loses information from the skool file: mostly formatting of my comments IIRC, but there was enough lost that as I got towards the end of my project that I stopped doing it.

If you're not sure how to modify a skool file, but you know how to make the corresponding modification to a control file (e.g. to replace a DEFM with a DEFB, as in your example), then using skool2ctl.py to go from skool to ctl and then back again is definitely the best way to go.

And there's no need to build a complete control file and then rebuild the entire skool file: skool2ctl.py has --start and --end options that enable you to carve out just the portion you want to modify. Then you can paste the resulting skool file snippet into your working skool file, instead of replacing the whole thing. The only caveat with that approach is the the --start address has to be the start address of a routine or data block.

[RMartins]

The CTL file looks like a blueprint of where code and data are supposed to be.

However, this is the first thing we NEED to do, which is most probably not the best scenario when we don't know anything about the code or data.

This would be a lot easier, if this representation of code/data (AKA CTL), was referenced but not replaced by SKOOL file.

And I say this, because disassembling something, is a process of discovery.
What this means, is that we should be able to go forward and back easily, which currently unfortunately is not that easy.

The DEFM is one of the cases, but there are others.
Blocks of that that now became variables
Blocks of code that became data.
Blocks of data that became code.

There is also the problem of bytes that are code and data, for example, when using self modifying code.

All in all, I see the CTL as a base reference to build a kind of node graph, that should be used by SKOOL file, but not to be replaced or embed its definition into the skool file. Because if the data is embedded, then we have issues to convert back and forth.

[RMartins]

This really needs some kind of editor, that would allows us to edit each block, brake it apart, join blocks, etc...

Because when we do these operations, we need to make decisions on how to merge or split and to keep or not the existing labels/comments/descriptions, etc ...
Because some have references to stuff that has changed.

Basically, this is similar to a resolve of merge conflicts in a Source Control system.

[RMartins]

By the way, using SkoolKit 8.6, I seem to have found a bug

This Macro doesn't produce the expected result (the output of the second third of the screen

Code: Select all

#SCR1,0,8,32,8(secondThird.png)

It comes close, but has a few problems:
- Some of the pixels and full lines, are missing
- Some colors Also seem off (but could be due to missing pixels).

Also tried to get the first third, but it has the same issues with pixel data.

Code: Select all

#SCR1,0,0,32,8(firstThird.png)

same problem for third, third.

Code: Select all

#SCR1,0,16,32,8(thirdThird.png)

[RMartins]

Found what appears to be another bug.
Maybe I should be reporting these elsewhere, but I'm not sure exactly were, so please enlighten me so that I can report them on the proper place.

So when processing "Formula One.TZX", the tapInfo.py correctly reports all the blocks.

Code: Select all

> tapinfo.py "Formula One.tzx"

Version: 1.10
1: Text description (0x30)
  Text: Created with Ramsoft MakeTZX
2: Standard speed data (0x10)
  Type: Header block
  Bytes: FORMULA 1
  CODE: 23296,1000
  Length: 19
  Data: 0, 3, 70, 79, 82, 77, 85 ... 232, 3, 0, 91, 0, 128, 76
3: Standard speed data (0x10)
  Type: Data block
  Length: 1002
  Data: 255, 62, 1, 50, 9, 92, 50 ... 0, 0, 0, 0, 0, 0, 49
4: Standard speed data (0x10)
  Type: Data block
  Length: 7202
  Data: 255, 0, 8, 8, 8, 8, 8 ... 0, 0, 0, 0, 0, 0, 90
5: Standard speed data (0x10)
  Type: Data block
  Length: 38936
  Data: 255, 205, 154, 233, 0, 0, 0 ... 96, 96, 96, 96, 96, 96, 129

As defined as supported by documentation, all data blocks are of type 0x10, except the initial description block (0x30).

However, when running tap2sna.py, it generates a very small (1279 bytes) but incomplete file.

Code: Select all

> tap2sna.py "Formula One.tzx" "F1.z80"

NOTE: I did unzip the file to avoid any possible issues while unzipping. But the result is exactly the same.

[RMartins]

The CTL file looks like a blueprint of where code and data are supposed to be.

However, this is the first thing we NEED to do, which is most probably not the best scenario when we don't know anything about the code or data.

...

All in all, I see the CTL as a base reference to build a kind of node graph, that should be used by SKOOL file, but not to be replaced or embed its definition into the skool file. Because if the data is embedded, then we have issues to convert back and forth.

Another issue that I have, is that we start with a CTL file, then lets suppose we edit the SKOOL file (either file will do).
So far so good, but now, we have another run of the game or we found an RZX, that we want to use to generate a new CTL file.

Even if we keep the SKOOL file to retain all the descriptions/editing documentation work, how are we going to merge this ?

Splitting it using start/end sections, generating a new distinct SKOOL file and editing/merging in the previous SKOOL file, all by hand ?

[SkoolKid]

Found what appears to be another bug.
Maybe I should be reporting these elsewhere, but I'm not sure exactly were, so please enlighten me so that I can report them on the proper place.

The place to report bugs is on GitHub.

However, when running tap2sna.py, it generates a very small (1279 bytes) but incomplete file.

Code: Select all

> tap2sna.py "Formula One.tzx" "F1.z80"

That's expected, because the TAP file contains headerless blocks. By default, tap2sna.py ignores headerless blocks because it cannot know where they should be loaded.

To make tap2sna.py load a headerless block at a specific address, use the '--ram load' option. Run

Code: Select all

tap2sna.py --ram help

for more info.

[SkoolKid]

By the way, using SkoolKit 8.6, I seem to have found a bug

This Macro doesn't produce the expected result (the output of the second third of the screen

Code: Select all

#SCR1,0,8,32,8(secondThird.png)

It comes close, but has a few problems:
- Some of the pixels and full lines, are missing
- Some colors Also seem off (but could be due to missing pixels).

[snip]

Can you share the skool file that you're using? I've just tried those #SCR macros on a loading screen of Hungry Horace, and they worked fine.

[SkoolKid]

Another issue that I have, is that we start with a CTL file, then lets suppose we edit the SKOOL file (either file will do).
So far so good, but now, we have another run of the game or we found an RZX, that we want to use to generate a new CTL file.

Even if we keep the SKOOL file to retain all the descriptions/editing documentation work, how are we going to merge this ?

Splitting it using start/end sections, generating a new distinct SKOOL file and editing/merging in the previous SKOOL file, all by hand ?

I'm not really sure what to say about all this, other than it sounds as if you're working on the skool file far too soon. It's a good idea to work on the control file exclusively in the beginning until you think you've identified all (or almost all) of the code, data and text. Then you can either carry on with the control file (as most SkoolKit users seem to prefer), or switch to editing the skool file, with the understanding that some manual conversion of code to data or vice versa might occasionally be necessary.

As I've said before, that's how I've done most of my disassemblies. I like editing the skool file directly because the instructions are right there, so it's easier to write annotations "in place". But that approach does require a decent familiarity with the skool file format, and perhaps that's why most people stick with the control file and its somewhat simpler syntax.