XT-Engine: small, but powerful platform game engine in asm

[ketmar]

moved some often used words to C kernel. XTE now compiles in 350 msecs. ;-)

p.s.: 250 msecs. ;-)

[ketmar]

and after some optimisation efforts we're at 190 msecs. no new asm parts moved to C, this is all low-level kernel and VM optimisations. ;-)

[ketmar]

sometimes it is very hard to stop… ;-) turned off some C kernel features i don't need, and implemented superinstruction optimiser. now we're at 170 msecs. i really believe that i'm done with it for now. because next step is native code generation, and i won't go that way. not yet.

superinstruction optimiser (read "peephole optimiser") doesn't really do much (it optimises a lot of instructions, but real speedup is barely noticeable), yet i simply wanted to implement it. no speedup is partially because i hand-optimised the code; but now i don't have to anymore. no more manual "2 +" to "2+" replacements and such: let the compiler take care of it!

also, "variable" and "value" access is as fast as simple direct memory access now: optimiser is smart enough to replace "var @" with one instruction which directly loads PFA of "var".

it's not a total waste of time, though: i believe that i will be able to use most of this code later, in Forth cross-compiler. this is also where i could implement my new idea of tracing native code generator too: it should generate machine code that does much less stack access than naive "subroutine stitching".

if my codegen idea will work as good as i expect it to, it would be possible to compile AGD-like to UrForth instead of to asm, and then cross-compile Forth to Z80 native code. but i'm not sure yet: there is simply not enough registers and addressing modes in Z80, and it may run out of registers too often, which will cause spilling. this needs a lot of R&D first.

[Wall_Axe]

yet: there is simply not enough registers and addressing modes in Z80, and it may run out of registers too often, which will cause spilling. this needs a lot of R&D first.

Why not design it for the z80...then just use a very similar system for windows and Linux. It would be (a tiny bit ) slow on modern systems but 100% compatible with the spectrum.

I think any compilation speed of two seconds or under is ok...
If you think about it, it takes two seconds to double click on the .tap file.
Fuse takes three seconds to start the game from double clicking.
So compilation speed isn't the bottle neck

[ketmar]

Why not design it for the z80...then just use a very similar system for windows and Linux. It would be (a tiny bit ) slow on modern systems but 100% compatible with the spectrum.

because there is simply not enough registers and address modes on Z80. ;-)

the main thing stack machine does is reading memory via indirect addressing. you cannot get rid of that, because this is how stack machine works. it can be workarounded on a CPU with many registers (RISC), or on a CPU with rich addressing modes (CISC, x86). and Z80 is neither.

Forth VM have two stacks, and there is simply no way to access both in the same time without performance penalty. there is no even "EX HL,SP" instruction to swith the stacks. and there is no way to use SP with displacement to access stack values. so either we have to POP everything (and there are not enough registers for that), or use IX as secondary stack pointer (and it would be slow enough to not bother with optimised native code generation).

this is, by the way, the problem with Z80 C compilers too: they have to do the same trick to access local variables. so good Z80 Forth with very simple peephole optimiser won't be slower than C compiler anyway. but i want it to be faster! ;-)

I think any compilation speed of two seconds or under is ok...
If you think about it, it takes two seconds to double click on the .tap file.
Fuse takes three seconds to start the game from double clicking.
So compilation speed isn't the bottle neck

i am using my asm not only to compile the code: most of the time i use it even to check the syntax. 1000+ compilations per working session is not something unrealistic, actually, i usually do more. so for me, compilation speed matters a lot. and i am not used to slow software. my code editor starts under 200 msecs, my emulator starts under 200-300 msecs, and so on. what most people see as "acceptable" (or even "good") i usually see as "slower than a dead snail".

anyway, the faster my Forth system is, the more things i can pack into my tools without being frustrated by slowness. ;-)

p.s.: and i cut XTE compilation time to 140 msecs. yay. ;-) it wasn't really a goal, i simply cleaned up UrForth code, reorganized it a little, and got a small speedup as a bonus.
p.p.s.: replaced OOP with simple words in label manager, now 120 msecs. OOP sux! ;-)

[Wall_Axe]

oh ok i stand corrected

does Abersoft Forth just handle it in a slow way? (much faster than BASIC of course)

BTW in Harry Potter they have a character named Aberforth Dumbledore. Was the author a fan of efficient programming languages for the ZX Spectrum?

[ketmar]

does Abersoft Forth just handle it in a slow way? (much faster than BASIC of course)

it doesn't use native code generation at all, it's all threaded code (indirect in the original, direct in dsForth). it's ok and not that slow, but i can do much better! they hadn't have much choice, though: proper optimising compiler would be so big that it will not leave much room for user programs. and Aberforth aimed to support tapes, so no disk swapping for turnkey app generation.

BTW in Harry Potter they have a character named Aberforth Dumbledore. Was the author a fan of efficient programming languages for the ZX Spectrum?

hehe. dunno, but i hope so! ;-)

p.s.: there were subroutine threaded code systems on ZX (i've definitely seen them, but don't remember the titles). it is "real machine code", but it wasn't much faster than Aberforth, and they used more memory for almost no real speed benefits.

[Ralf]

My feeling is that too much steam is going into the whistle in this thread.

It really doesn't matter if something compiles in half a second, one second or three seconds.

By the way, do I understand correcly that game compiler running in Forth runs on actual Zx Spectrum?

Personally I would go 100% cross development path and forget any efficiency issues once forever but it's your decision Ketmar.

[ketmar]

My feeling is that too much steam is going into the whistle in this thread.

i am working on UrForth anyway. i could be silent about it, but that wouldn't change the fact that i'm working on it. ;-)

XTE2 itself is finished (in the sense that it is ready to use), but it is missing almost all the tools required to build a game. that's what i am actually doing right now: developing game creation tools. fast Forth system is essential here, because all the tools will be written in Forth.

It really doesn't matter if something compiles in half a second, one second or three seconds.

it does for me. think of it as one of my personal quirks. ;-) i strongly believe that computer programs should be FAST. most of apps these days are so slow that they are almost unusable. i hate it (that's why i wrote a lot of apps for myself from the scratch, and keep doing it — this includes almost the whole range of apps, from im and e-mail clients to text editors).

besides, asm translator doesn't do any sohpisticated work, yet it was doing it in almost half a second. now, add the sophisticated compiler from the high-level language on top of that, and the speed will go down fast. 460 msecs to 120 msecs might not look impressive, but it actually means that i managed to make the whole system almost 4 times faster. now, "4 times faster" looks more impressive, isn't it? ;-)

By the way, do I understand correcly that game compiler running in Forth runs on actual Zx Spectrum?

oh, no! it is the system running on PC, and it does cross-compiling. Speccy is simply not powerful enough. i want to have AGD-like system, but as a cross-compiler. you do everything on PC, and build the game on PC.

p.s.: i prolly should ask Peter to change the thread title, it is somewhat misleading right now. XTE project mutated from "asm game engine nobody will ever use" to "XTE: the fully-featured game cross-development system, powered by fast and small native ZX Spectrum game engine (and Forth) which nobody will ever use". ;-)

[Stefan]

My feeling is that too much steam is going into the whistle in this thread.

I’m really enjoying reading about the steam.

[ketmar]

I’m really enjoying reading about the steam.

that's also how i am trying to keep the project alive. ;-) writing C code is boring; i started AGD-like compiler in C, but never finished it. booooring. using Forth is way more fun, and the project has much bigger chance to succeed this way. XTE1 stopped exactly at the game creation tools programming stage. let's see how XTE2 will do. ;-)

[ketmar]

i hope you all thought that i am working heavily on AGD-like, or room editor, or something like this. because of course i am not! i mean, i started but… but i suddenly realised that trying to keep UrForth/C as "embeddable scripting engine" is stupid idea. it has no app to be embedded into anymore: UrAsm is Forth app now, not vice versa.

threaded code adventures

so i replaced internal memory model which was using virtual addresses with real addresses (no more expensive checks on each access), and now replacing threaded code with primitive-centric DTC. spent two days trying to convince GCC stop breaking my code. this is "almost native system" now, only without using asm. i'm not sure if it worth it, though, because i'm still fighting with GCC, and unable to run tests. if it will not bring at least x5 speedup, i will prolly throw it away. but it was totally worth trying, and it already benefited the project: i nailed (let's hope ;-) compiler design, so i don't have to change Forth code when i change the kernel anymore. it means that i can switch kernels back and forth (oops) now, using any kind of threaded code i want, and don't rewrite anything. which means a lot, because most of the system (including the compiler) written in UrForth itself.

and if you think that i am obsessed with minor speed improvements instead of creating The Real Thing first… well, you may be right. but it almost physically hurts me when i know that i can do much better, and it costs only several days of development. i mean, if we will take GCC out of the picture, i didn't knew that GCC is so broken stubborn.

[Wall_Axe]

Why don't you use x86 or x64 ASM compiler to make your forth

[ketmar]

Why don't you use x86 or x64 ASM compiler to make your forth

because i'm not making a Forth system, i am writing Forth system to implement other parts of XTE project. so i need UrForth/C to run on various architectures without rewriting the kernel. it should work this way: you download the sources, you compile C kernel, everything else Magically Works. and i don't have to port the system to, for example, ARM: C compiler will do it for me.

wall of text, trying to explain things and failed ;-)

i already have UrForth/x86 — this is native x86 system, written in assembler and itself, and it doesn't require anything else to work, or to be rebuilt. it includes its own assembler, so i don't need any external tools. ;-) i am GNU/Linux user, i have my 32-bit system, and no plans to switch to 64 bits. so i'm ok with that. but other people may use other OSes, and vendors insist on 64-bit idiocity, so users have almost no choice. i don't expect people who may want to try XTE to port my native Forth system to their OS first. ;-)

there is nothing hard (and nothing wrong) in writing new Forth system suitable for some task. actually, that's how Forth is used. what you see in this topic is a gradual rewrite of the whole project: UrAsm written in C with Forth scripting becomes Forth kernel written in C with UrAsm as Forth application on top of it. it already works, i'm simply not satisfied with 120 msecs to compile XTE Z80 sources, i want it faster. hence i already rewrote Forth kernel several times, making it faster with each rewrite. UrAsm code is not changed, only the Forth kernel.

i started with slow, but working system, and now optimising it. most of the time people simply don't have a chance to see it happens in real time. ;-) i could disappear for month or two, and then pop out with rewritten thing, and nobody would knew that it was started with "half a second to compile XTE", and finally was optimised to "80 msecs to do the same". XTE itself was developed like that too: barely working slow and glitchy code, which was optimised to something usable. only with Forth i chose to do it in blog-like format, because… because why not? ;-)

let me explain why i'm trying to make it fast with slightly more details. ;-) it is known thing (scientific! ;-) that people start noticing program delays when those delays take more than 200 msec. i.e. if something works in 180 msecs or less, most people see it as "instant". this also works for me. so 460 msecs was unacceptable: while technically it is very fast, psychologically it is slow. i managed to reach 120 msecs, but… there is a problem: it is dangerously close to 200. as i'm writing more tools (sprite and room converters, high-level language compiler, and such) it will eventually became slow again. and i know for sure that some changes in the interpreter will allow me to cut everything to 50-80 msecs (at least). which gives me plenty of room (time ;-) to stop worrying about the speed.

as i am planning to use XTE myself, it is important for it to not trigger me each time i'm compiling the code. i already added some tools, and it became slow again. i would accept it, but… but i KNOW the way to make the system much faster. my lazyness is not the excuse here. ;-)

[ketmar]

the new UrForth/C engine is up and running. XTE compile time: 40 msecs. ok, it's not 5 times, but a cat 3 times is fine too. ;-)

fun thing: on "pure-forth" benchmarks UrForth/C is faster than native UrForth/x86. mostly due to different threaded code model, of course, and switching native version to this kind of threaded code will make it faster than C kernel again. something i'll do in the (far) future.

[Wall_Axe]

Ok glad you got it working,it seemed like the C compiler was not working for you,but it is.

[ketmar]

Ok glad you got it working,it seemed like the C compiler was not working for you,but it is.

thank you. tbh, i had to submit to GCC demands, and write my code the way it wants, not the way i want. ;-) to be even more honest, i simply wasted two days trying to fight GCC instead of trying to use it. converting the kernel to proper code took less time than trying to force GCC to do strange things.

still, it may be looked at as "asm version". because what i am doing everywhere is casting pointers to 32-bit integers and back, so the end result as unportable as if i would write it directly in asm. lol.

i decided to ditch 64 bits because… because there is no way i can properly support it and stay fast anyway. my Forth code expects everything to be kept in 32-bit cells, including addresses. and i am using pointers a lot. there is "CELL" word which holds cell size, and it is even used in the code… but there are also "8+" to skip two cells everywhere too. making the kernel work with 64-bit system is easy, but debugging Forth code is… well, absolutely impossible, because i simply don't have 64-bit system to debug on.

anyway, 32-bit version will always be faster than 64-bit one, so there is no real reason to bother.

p.s.: the system is almost IDE now. ;-) this is because Forth is interactive at its core, and i quickly wrote a simple fullscreen text editor for it, so i could test my new words without leaving the system. it is ugly, it will never be published (at least in its current form), but it works. oh, yeah, it took hour or two to write, so you can imagine how good it is. ;-)

[ketmar]

and by the way: Fossil repo for the new UrForth/C engine. there's nothing really to look at, but if you are curious…

[ketmar]

talking about Forth is fun, talking about other things is not. that is, i am working on boring things like room design tools. which mostly consists of creating wrappers for Xlib (or using terminal — that's what i'm doing right now). as the design of this thing (room management) is not very clear for me, i am in research phase again. hence updates will not be as frequent as they were.

Spoiler

actually, i am writing integrated UrForth text editor, so i could do most things without leaving the system. i also need simple GUI for sprite and tile editors, and room editor with tile view. it is mostly a boring task of writing high-level GUI libs.

just in case: nope, it wouldn't be easier with any other language. i am always writing GUI libs from scratch for each my new project. no joking. so it doesn't matter if language X already has GUI tools, i won't use them anyway. this time i am also writing the compiler from scratch, but it's not the first time i'm doing that too. ;-)

[ketmar]

AGD-like compiler needs asm! i mean, yes, we already have a working assembler, but it is absolutely not fun to spend time building asm command strings for UrAsm to immediately parse them back again. so i wrote Forth-style Z80 assembler too, which is much smaller, somewhat faster, and way easier to use in codegen. it also can be used alongside UrAsm. so now we have two Z80 assemblers!

talking about asms

why writing a second one, again? see, to generate code with UrAsm, i need to first build a string like "ld hl,69", and then pass that string to UrAsm. then it will be parsed, and Z80 code will be generated. and i know what kind of opcode i want, so it is a wasted time and resources.

of course, i can use direct code generation with machine code values: `0x21 c, 69 c, 0 c,`. this is fast, but totally unmaintainable. i have both to remember what instruction "0x21" is, and carefully check if i wrote it right.

or, i can use Forth-style assembler, and the thing will look like: `ld hl, # 69`. no, this is not yet another string, this is actual Forth code, which is compiled and executed as any other normal Forth code. it is fast, it is readable, it is easy to write. and the very same thing can be used as "normal asm", with some small syntactic quirks. like:

Code: Select all

ld a,69  →  ld a, # 69
ld hl,(1234)  →  ld hl, () 1234
ld (ix+7),42  →  ld 7 (ix+), # 42 (or ld (ix+) 7 , # 42)

"a," and other alikes are words, without space before the comma (but "a ," is allowed too). and i need to tell the assembler what the number means: "#" for numeric value, or "()" for memory reference. this is somewhat unusual, but not something terrible.

note that this code can be used as-is in codegen too. as command arguments are Forth words, i can dynamically build instructions, like: `ld use-de? if de else hl endif , 69`.

ah, and by the way. UrAsm source is 100+ kb. Forth-style asm source is ~20kb. yes, 20kb, because i only need to perform several very simple numeric checks to catch errors, and then several shifts and ors to build the opcode. it can be ~10kb if i'll remove some optional features and most error checking, but hey… 20kb is not that much! ;-)

that's what you can get if you stop fighting with the language, and start using it as intended.

this asm will be used in some other my projects too.

[ketmar]

by the way, Forth-style Z80 asm source. yeah, that's all of it. nothing really complicated, just "check operands, generate code". actually, operand descriptions is the longest part (and it simply builds small words, so compiled code is much smaller than the descriptions themselves).

it took me a day to write. less than a day, actually, i procrastinated a lot. once you decided how to build instructions, and designed data structure for instruction builder, everything else becomes trivial. the core of the asm is "a;" word, which generates instruction code using the variables ("quan"s) above. this word basically IS the assembler. everything else is just a boring code to fill the variables. ;-)

[Wall_Axe]

What does it do

[ketmar]

exactly what other Z80 assemblers do: convert asm mnemonics to machine code. ;-)

[Wall_Axe]

So you're progressing with the new system

[ketmar]

yes. it may not look like it, but i'm slowly moving forward. this new asm, for example, is useful in compiler code generators (and that's why i created it — for codegen). UrForth/C is a very mature system now too (and it is faster than my older UrForth/x86 native, lol). i will also got rid of C kernel eventually. you were right asking why i didn't wrote it in asm in the first place! ;-)

it may look like i'm mostly working on UrForth… and it is definitely so. i am basically creating the whole toolchain from scratch, starting with Forth compiler and up to text and gfx editors. tbh, the whole idea of XTE project is to have some fun; XTE dev system itself is a side product of me having fun. ;-) so i decided to do it the way we were doing it 20+ years ago: write my own compiler, write my own operating shell, my own editors, and so on. and i really love to use Forth, but the idea of using it in other projects always shot down with "where we'll get a second competent UrForth programmer?" question. ;-)

as i wrote somewhere above, Z80 part of XTE is mostly complete (it can be optimised a little here and there, but it is feature-complete for now; well, except per-pixel collision detection routine). now i am trying to create a "development studio" for it, with editors, cross-compilers, and such. it will be a highly flexible environment, because the user could use the same UrForth language to extend anything. but i am not really expecting people to learn UrForth, so i need to make everything usable out of the box.

yes, i deliberately made the whole task harder than it needs to be by not using, for example, C++ and Qt. but this is how i am having fun: doing things from scratch, in a language nobody else is using. ;-)

p.s.: "learning UrForth" is not a typo, because UrForth is written without looking at any "Forth standard". i strongly believe that all Forth standards sux, i and know better. so knowing, for example, ANS standard won't help one to read something like "compiler:comp? ?< ['] forth::(#) <\, , \> \\ . || . >?". yes, this is actual working UrForth code, not some random line noise. ;-) and it is perfectly readable when you learn some system naming conventions: all those punctuation marks built with several universal patterns. (if there is somebody with Forth knowledge here, try to decipher it. you should succeed even without a manual.)