The eternal quest for the pristine snapshot

[SkoolKid]

Probably one of the most frequently asked questions about SkoolKit is whether its tap2sna.py command can reliably convert a TAP or TZX file into a "pristine snapshot", i.e. a snapshot of the game at the point where it has finished loading and is just about to start running.

Up till now, the answer to that question was: Well, maybe, but only if the TAP/TZX file contains no non-standard blocks (i.e. blocks loaded by anything other than the ROM load routine) or headerless blocks, and if you're also willing to put in the time to figure out what the game's start address is.

So I'm pleased to announce that tap2sna.py just got a bit more reliable. It still requires standard speed, ROM-loaded blocks, but it can now handle headerless blocks, and with any luck it will also automatically figure out the game's start address. If you want to try it out, you'll need to grab the latest development version of SkoolKit from GitHub (either by cloning the repo, or by clicking the green Code button and selecting Download ZIP). Once you've done that, find a TAP file, and do the following:

Code: Select all

tap2sna.py --sim-load game.tap game.z80

The new --sim-load option simulates a freshly booted 48K Spectrum running LOAD "". Here it is working on a TAP file for Manic Miner:

Code: Select all

$ tap2sna.py --sim-load manic_miner.tap manic_miner.z80
Program: ManicMiner
Loading data block: 23755,69
Bytes: mmm       
Loading data block: 22784,256
Bytes: mm1       
Loading data block: 32768,32768
Simulation ended: PC=33792
Writing manic_miner.z80

As you can see, it's correctly determined the start address: 33792.

Anyway, this functionality will be present in the next full release of SkoolKit (8.7).

As for non-ROM-loaded blocks, turbo loaders, and other custom loading schemes, well, I'll probably be looking at supporting those for SkoolKit 8.8. All in good time.

[Bedazzle]

For me it is more sane to load game in emulator supporting tzx, and put breakpoint to screen area write (usually games after loading do screen clear).
Then just dump whole 48K dump to a binary file. At this point you know what is "starting" address and where is stack, and interrupts.

Thus way it does not bother you, if tap/tzx does contain weird blocks, or not, and if it does some decrypting or backward loading.
Yes, there are loaders, that use rom timings, but load file in non-standard way.

[Lethargeek]

i use "exit subroutine" button in SPIN debugger
then trace the PC until it points into the RAM
(trapping USR ROM routine for BASIC loaders)
it works for non-ROM loaders as well

and just to be sure nothing below the stack is corrupted
trace until the stack pointer moves for 48k snapshots

[SkoolKid]

Using an emulator as you've described is all well and good, but it's not a technique that lends itself to scripting and automation. That's what I'm aiming for with the --sim-load option of tap2sna.py.

By the way, there is a macro front-end to the Z80 instruction set simulator in SkoolKit 8.7: #SIM. I haven't played much with it myself yet, but I'm thinking it might be useful for creating images or screenshots that would be difficult to do without creating a SkoolKit extension or some wizard-level macro definitions: just simulate execution of the code that draws everything you want to the screen, and then invoke the venerable #SCR macro. Done.

[SkoolKid]

Just did some playing with #SIM, and here's a fun example:

Code: Select all

@rom
; Welcome
;
; #SIM$129C #SCR
c32768 RET

Can you guess what that #SCR macro creates a screenshot of?

[Lethargeek]

Using an emulator as you've described is all well and good, but it's not a technique that lends itself to scripting and automation. That's what I'm aiming for with the --sim-load option of tap2sna.py.

But you may apply the same logic to your script, not limiting it just to the ROM loaders.That is, when the tape ends, just skip until the PC points into the RAM. And for 48k sna, skip more to before-PUSH or after-POP as a precaution.

[SkoolKid]

That is, when the tape ends, just skip until the PC points into the RAM.

--sim-load already does that. The reason it only works for ROM-loaded blocks at the moment is that it intercepts calls to the ROM load routine. It doesn't (yet) convert TAP/TZX blocks into a sequence of port 254 readings for a custom load routine to consume. But as I said, all in good time.

[crabfists]

This is great! Thanks for adding this feature. Can't wait to try it out.

I recently spent a few hours making a pristine snapshot for Auf Wiedersehen Monty. I ended up using an emulator to see where in memory the data was being loaded and then created a t2s file from the findings. Was a bit long-winded but it worked. The loader for Auf Monty uses a modified version of the LD_BYTES ROM routine. It calls LD-EDGE-1 and LD-EDGE-2 directly. Any idea if that will still work with your --sim-load option? Or will it only work if LD_BYTES is called directly?

[SkoolKid]

The loader for Auf Monty uses a modified version of the LD_BYTES ROM routine. It calls LD-EDGE-1 and LD-EDGE-2 directly. Any idea if that will that still work with your --sim-load option? Or will it only work if LD_BYTES is called directly?

At the moment --sim-load will only work if LD_BYTES (at 0x0556) is called directly. If you're familiar with Python, you can see the code that intercepts (and bypasses) LD_BYTES here:

https://github.com/skoolkid/skoolkit/bl ... na.py#L172

In due course I'll be working on support for custom loaders.

[crabfists]

That's great it will support custom loaders in the future. I admire your dedication.

In a similar search for an automated way to make pristine snapshots I modified tapinfo.py to iterate through all the blocks in a tape file searching for the block's data in a given z80/sna file. The idea was it would tell you where the blocks ended up in RAM. The tool turned out to not be that useful but it was a good learning exercise. It worked for some games but would fail more often than it succeeded. Your idea of actually doing the loading is much better.

[Bedazzle]

That is, when the tape ends, just skip until the PC points into the RAM.

It is not always good. For example, "Mask 3: Venom strikes back" have additional file for 128K loading (main menu AY music). So, 48K does not load this part.

[SkoolKid]

As promised, I've added support to tap2sna.py for custom loaders (including turbo loaders). Here's --sim-load working on a TZX of Skool Daze (which has a huge turbo load block):

Code: Select all

$ tap2sna.py --sim-load --start 24288 skool_daze.tzx sd.z80
Leader tone
Sync pulse
Data
Loaded bytes: 23778,17      

Leader tone
Sync pulse
Data
Loaded bytes: 23755,331     

Leader tone
Sync pulse
Data
Loaded bytes: 16384,82107   
Simulation ended: PC=24288
Writing sd.z80

The --start option is needed here because Skool Daze doesn't actually load every byte that's saved on the tape, so you need to tell the simulator when to stop (by telling it where the game starts).

The old --sim-load, which only works with tapes that contain ROM-loaded blocks, is now available as --sim-load-fast. As the name suggests, it is faster (MUCH faster) than --sim-load. In fact, on my 8th gen. core i7, --sim-load operates at about 60% of the speed of a real Spectrum. Clearly the Z80 simulator needs some optimisation for speed.

By the way @crabfists, I tried --sim-load on a TZX of Auf Wiedersehen Monty and it worked like a charm. (A slow charm, that is.)

[Bedazzle]

Got fresh SkoolKit from Github, and tzx from SC (
https://spectrumcomputing.co.uk/pub/sin ... ze.tzx.zip

then run
tap2sna.py --sim-load --start 24288 "Skool Daze.tzx" sd.z80

It hung in loop, numbers are running till 65535, then from 0 to 65535, and again from 0 to 65535...

[SkoolKid]

Got fresh SkoolKit from Github, and tzx from SC (
https://spectrumcomputing.co.uk/pub/sin ... ze.tzx.zip

then run
tap2sna.py --sim-load --start 24288 "Skool Daze.tzx" sd.z80

It hung in loop, numbers are running till 65535, then from 0 to 65535, and again from 0 to 65535...

You need to be more patient!

The Skool Daze turbo loader loads 82107 bytes in total, spanning both the ROM and the RAM, with the load address incrementing by 23 after each byte has loaded. That's why it looks like an infinite loop - but appearances are deceptive.

By the way, it's occurred to me that --sim-load should stop if a custom loader has been running and the program counter suddenly hits an address in contended memory (where no respectable custom load routine would reside). That way there would be no need to specify --start for Skool Daze (for example).

[Bedazzle]

You need to be more patient!

Oh, boy. It worked whole 15 minutes, and finally did it.

[Lethargeek]

It is not always good. For example, "Mask 3: Venom strikes back" have additional file for 128K loading (main menu AY music). So, 48K does not load this part.

well, nobody is stopping you from saving sna every time the condition is met and then pick the last one before the game starts

[SkoolKid]

Oh, boy. It worked whole 15 minutes, and finally did it.

Oof! If you have the time, try running it again with the latest tap2sna.py. I've just made some performance improvements that cut the time from 12m10s to 6m30s on my computer.

[Bedazzle]

Oof! If you have the time, try running it again with the latest tap2sna.py. I've just made some performance improvements that cut the time from 12m10s to 6m30s on my computer.

It loads in 9 minutes now.

[SkoolKid]

For those subscribed to the --sim-load newsletter: I've now added support for TZX block types 0x12/0x13/0x14. In plain English that means about 99% of all TZXs out there should work (in theory).

If you're looking for a TZX that contains block types 0x12/0x13/0x14 to try with --sim-load, I suggest 'The Great Escape.tzx', which uses Speedlock IIRC.

[SkoolKid]

Time for a performance review. If you're sick of writing novels, building houses from scratch, or placing yourself in cryogenic stasis while tap2sna.py --sim-load loads a tape (and who isn't?), then this is for you.

Here's a list of times taken by the command

Code: Select all

tap2sna.py --sim-load --start 24288 skool_daze.tzx sd.z80

with various versions of SkoolKit on various dates on my desktop PC:

• 4m10s - SkoolKit 8.7 (2022/10/08)
• 1m07s - SkoolKit 8.8 beta (2022/11/03)
• 0m30s - SkoolKit 8.8 beta Cythonized (2022/11/03)

So still not blazingly fast - spoiler: a Z80 instruction set simulator implemented in pure Python never will be - but much improved since SkoolKit 8.7.

[SkoolKid]

More times taken by the command

Code: Select all

tap2sna.py --sim-load --start 24288 skool_daze.tzx sd.z80

running on my desktop PC:

• 1m06s - SkoolKit 8.8 beta, Python 3.9 (2022/11/05)
• 0m41s - SkoolKit 8.8 beta, Python 3.11 (2022/11/05)

Yes, Python 3.11 really is that much faster than Python 3.9 (and Python 3.10, btw). Upgrade now for faster --sim-load action!

[SkoolKid]

I've just added support for tape-sampling loop accelerators in SkoolKit 8.9 beta. So far the only accelerators I've actually implemented are 'microsphere' and 'rom', but more are on the way.

Here are some --sim-load times for Skool Daze using Python 3.11, with and without accelerators (for comparison):

Code: Select all

tap2sna.py --sim-load --start 24288 skool_daze.tzx sd.z80                           # 0m36s
tap2sna.py --sim-load --accelerator microsphere --start 24288 skool_daze.tzx sd.z80 # 0m16s

And for Contact Sam Cruise:

Code: Select all

tap2sna.py --sim-load contact_sam_cruise.tzx csc.z80                   # 0m43s
tap2sna.py --sim-load --accelerator rom contact_sam_cruise.tzx csc.z80 # 0m11s

[SkoolKid]

Just added support for accelerating Speedlock and Firebird BleepLoad, and therefore any other loader that has the same tape-sampling loop as either of those. In other words, the following accelerators are now available:

• bleepload (Firebird BleepLoad)
• cyberlode (Cyberlode 1.1)
• elite-uni-loader (Elite Uni-Loader)
• excelerator (The Excelerator Loader)
• ftl (FTL)
• gargoyle (Gargoyle)
• injectaload (Injectaload)
• microsphere (Back to Skool, Skool Daze, Sky Ranger)
• power-load (Power-Load)
• rom (any loader whose sampling loop is the same as the ROM's)
• speedlock (Speedlock - all versions)
• zydroload (Zydroload)

On my machine using Python 3.11, the speedlock accelerator cuts the tap2sna.py --sim-load time for Bruce Lee from 23s to 8s, and the power-load accelerator cuts the LOAD time for Dynamite Dan from 21s to 9s.

[SkoolKid]

I've now implemented accelerators for every tape-sampling loop listed on this page (except Alkatraz, for technical reasons).

Does anyone know of any other tape-sampling loops that are used out there in the wild?

[SkoolKid]

A quick tap2sna.py update. It's been a while. In SkoolKit 9.0 (currently under development), tap2sna.py:

• can create 128K snapshots
• has a phantom typist (so can load games that require something other than plain old LOAD "" or LOAD ""CODE)
• can create SZX snapshots (useful for at least two games)
• does a simulated LOAD by default (no more --sim-load option)

For the uninitiated: tap2sna.py is a tape-to-snapshot converter. For many games it works as simply as this:

Code: Select all

$ tap2sna.py https://spectrumcomputing.co.uk/pub/sinclair/games/b/BruceLee.tzx.zip
Downloading https://spectrumcomputing.co.uk/pub/sinclair/games/b/BruceLee.tzx.zip
Extracting Bruce Lee.tzx
Program: Bruce Lee 
Fast loading data block: 23755,1421
Data (21 bytes)
Data (43117 bytes)
Tape finished
Simulation stopped (end of tape): PC=60513
Writing Bruce Lee.z80