Working around the memory limits

[alban lusitanae]

HI Forum

In my return to trying some programming for fun, I have come up with some crazy ideas.
Ideas which of course will make my available memory run out pretty fast.

I have a project which will require large amounts of non-repeatable text to be printed to the screen,
although almost always at the same location in the program.

I first thought (considering the majority of people use emulators with fast loading anyway)
to use MERGE at the specific time to bring the large amount of text I need
overwritting the same lines of code I want for those large chunks of different text (more or less the same size),
in a way that the other code remains exactly the same and just loads pages and pages of info.

The problem is that if I am reading right how merge works, it will work perfectly in overwritting just the lines of code I need,
but will not load to / use the same memory, it will just keep adding up and using the available one
(even though overwritting the same lines on the program) until I eventually Code 4 the program.

Any thoughs on how to work around this thing?

Thanks in advance
Alban

[andydansby]

The first question is are you developing in Sinclair BASIC or are you planning on using another language?

[alban lusitanae]

Basic only for now, hve no time to learn Assemb or Machine

[alban lusitanae]

dont think it is anything fancy, just doing things for fun

[andydansby]

Basic only for now, hve no time to learn Assemb or Machine

There are a few alternatives to doing it in pure Sinclair Basic. First try this:

https://zxbasic.readthedocs.io/. It a pretty solid cross compiler using BASIC.

If you feel like using an alternate to BASIC, you may want to give Z88dk a try https://z88dk.org/site/. You can within Z88dk use a compression engine such as the impressive ZX0 engine.

If you're planning an adventure game, there are a number of open source codes that allow you to write one in C.

Cheers

[WhatHoSnorkers]

Would saving the strings as an array work, and then loading them?

So if you have 10 lines of text, DIM T$(10,32), then SAVE "text1" DATA t$()

and then loading the appropriate data. Won't be fast of course.

[catmeows]

You can save some space by replacing common digraphs by unused character codes. It doesn't compress text as well as better methods but it still can offer 90% - 85% ratio (i.e. saving 2 or 3 KB on 20KB text).

[flatduckrecords]

The manual asserts that the data is deleted, but you need to have enough free memory to hold the old data until the new data is fully loaded and they can be 'swapped'. I don't believe it will grow cumulatively with subsequent LOADs or MERGEs.

MERGE only deletes an old program line or variable if it has to because there is a new one with the same line number or name.

LOAD name DATA letter$()
LOAD name DATA letter()
deletes any array called letter or letter$ (as appropriate) and forms a new one from the array stored on cassette.

Error 4 Out of memory occurs unless there is enough room in memory for all of the old program and variables and all of the new program and variables being loaded from tape.

I think it might work well if you load the data in smaller chunks (so that the amount of temporarily "overlapping" memory is limited). A combination of @WhatHoSnorkers' arrays and @catmeows' compression sounds good to me!
EDIT:

However

Error 4 out of memory occurs if no room for new arrays. Old arrays are not deleted.

But with MERGE:

Error 4 Out of memory occurs unless there is enough room in memory for all of the old program and variables and all of the new program and variables being loaded from tape.

So I think what I said before is true of program lines (i.e. they are loaded-then-swapped) but perhaps not true of arrays after all!

[AndyC]

It won't use the same memory, in that it will likely reallocate things but I don't think it retains the old lines once it has replaced them as that wouldn't make a lot of sense. It might need enough work space to hold both at once though as I guess it doesn't know which lines to delete until it has loaded the new code.

I suspect loading an array would be more efficient, since presumably BASIC will release the memory used by the array currently before loading anything, although I can't say I've ever looked that closely.

[Hedge1970]

Have you thought of using the Microdrive as a database https://spectrumcomputing.co.uk/entry.php?id=2000231

[Ralf]

I strongly encourage you to start using asm. It'll be hard at beginning but when get it, even in some limited way, you won't ever look back to Basic. For some code complication you will get the freedom to do anything you like the way you like + speed that you'll never achieve in Basic.

For now you can try to save your texts as binary block of code.

Then you can load many blocks to the same place:

LOAD "TEXT1" CODE 40000
LOAD "TEXT12 CODE 40000
...

The problem is that you won't be able to display it with simple PRINT command.

You can try a loop to print it one character by one. But it can be slow unfortunately.
So maybe a small asm routine?

[Joefish]

You could build a dictionary of words, or write your own form of shorthand where you use some of the keyboard symbols that you'll never normally type as substitutes for common words or sets of letters, e.g. '@' represents 'a' followed by a space. '#' for 'the' + space. Look at what you've written for the most common words or even parts of words which can be picked out and replaced with symbols to save memory. Simply replacing every 'th' with '+' or 'sh' with '$' will save you a byte every time it's used. The text might not be easily readable to anyone else, but you'll quickly learn to write using your shorthand. Just so long as the code to expand the text again is short and efficient, so that it doesn't waste all the memory you saved. Scan each character one-by-one as it's printed, and expand any special symbols into whole words.

You could type everything in lower case but have a special code every time you need a capital letter, as they're quite rare in normal writing. e.g. '^a' means 'capital A'. That way, all capital letters now become available as ways of encoding whole words in shorthand, just like the Spectrum's keyword system. You then have another 26 common words that can be represented in your main text as a single byte.

If you were using machine code, there are all sorts of more efficient storage systems you could use, but these are ones you could implement fairly easily in BASIC. The trick with compression techniques is to only encode the things you re-use at least three times (as a general rule), otherwise storing the 'dictionary' version of your data, it's compressed form, and the decompresser function are likely to take up more memory than the original information. But if you can compress something you use dozens or hundreds of times, the benefits really start to add up. Small words like 'the ', 'a ', 'of ', and the names of common scenic features are a good place to start.

[p13z]

Quick thing to mention, too:
I don't think there is an easy way to continue/autorun using MERGE - beyond an ASM routine handling error trapping.
(The program will stop and need a RUN or GOTO to continue after a MERGE command is executed).
If I was dong something along these lines - I'd consider writing the text as a .txt file, and dumping it into Speccy memory above RAMTOP. Then it can be loaded/saved using CODE, and read in BASIC with PEEK.

[flatduckrecords]

I was curious about the "old arrays are not deleted" thing from the manual so I did a quick test using Alessandro's tip for calculating free memory ("65536-USR 7962"), and loading three data files (data1, data2, data3):

Code: Select all

10 DIM t$(10,32)
20 FOR d=1 TO 3
30  GO SUB 80: GO SUB 90
40  LET f$ = "data" +STR$ d
50  LOAD f$ DATA t$(): REM t$() will be resized depending on the data
60 NEXT d
70 GO TO 20
80 PRINT #0;AT 0,0;"Free bytes: ";65536-USR 7962: PAUSE 0: RETURN 
90 PRINT #0;AT 0,0;"Loading #";d;"...     ": PAUSE 0: RETURN 

And it seems replaced arrays are not left floating around in memory, and redimentioning an array to a different size consumes more memory (or frees memory up) as you'd expect according to the new size. This doesn't show memory used durning loading of course, but it means repeatedly loading the data doesn't waste memory in a leaky way so I think this technique is fine @alban lusitanae (if not optimal, as others have noted).

[alban lusitanae]

Thank you very much to all of the ideas... a LOT to consider and think about for sure....

[alban lusitanae]

I was curious about the "old arrays are not deleted" thing from the manual so I did a quick test using Alessandro's tip for calculating free memory ("65536-USR 7962"), and loading three data files (data1, data2, data3):

Code: Select all

10 DIM t$(10,32)
20 FOR d=1 TO 3
30  GO SUB 80: GO SUB 90
40  LET f$ = "data" +STR$ d
50  LOAD f$ DATA t$(): REM t$() will be resized depending on the data
60 NEXT d
70 GO TO 20
80 PRINT #0;AT 0,0;"Free bytes: ";65536-USR 7962: PAUSE 0: RETURN 
90 PRINT #0;AT 0,0;"Loading #";d;"...     ": PAUSE 0: RETURN 

And it seems replaced arrays are not left floating around in memory, and redimentioning an array to a different size consumes more memory (or frees memory up) as you'd expect according to the new size. This doesn't show memory used durning loading of course, but it means repeatedly loading the data doesn't waste memory in a leaky way so I think this technique is fine @alban lusitanae (if not optimal, as others have noted).

Well even if they are like 20 blocks of full screen text being loaded from tape each time I need, I think I have enough space to allow it to load (heck I will use 128k basic if I have to)

[Joefish]

If I was dong something along these lines - I'd consider writing the text as a .txt file, and dumping it into Speccy memory above RAMTOP. Then it can be loaded/saved using CODE, and read in BASIC with PEEK.

This is the best idea if you are going to do a multi-load. Use CLEAR to lower the RAMTOP (which does shrink the BASIC area), but POKE or load the text into the top of memory in an emulator. Either write down for use in BASIC, or also POKE in, a table of memory addresses where each bit of text starts.

Again, you could also use the upper ASCII codes (127-255) as codes for common words (the way BASIC encodes keywords). But for that, you'd need to write some code outside of your main program to prepare and encode it all for you. It's all in the preparation! And how much pain are you willing to suffer for your art?

[p13z]

Quick thing to mention, too:
I don't think there is an easy way to continue/autorun using MERGE - beyond an ASM routine handling error trapping.

I just checked this, and it appears I'm mis-remembering something. The Speccy is happy to continue running after a MERGE in a listing.