How to encode large amounts of text/data.

The place for codemasters or beginners to talk about programming any language for the Spectrum.
Post Reply
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

How to encode large amounts of text/data.

Post by Nomad »

I was thinking of better ways to encode text data into a game, without having to go the hobbit route..

Assuming you have a game were there is a large amount of repeated text, but you want to display it in a series of different arrangements. Like for example in elite. You have the planet descriptions. That is probably the best example I can think of.

Key for the tables:

Code: Select all

A Animal to see (zoo or hunting)
B Boring/Tedious
C frequent/deadly/dreadful Civil war
D killer/evil/lethal Disease
E occasional/unprEdictable civil war
F great/vast Forests
G plaGue of killer/deadly animals 
H Hoopy casinos (see 1)
I Unusual/Weird Forests (Indigo coloured leaves)
J Juice (from animal/tree)
K sports (Karate) that need gravity (Championships, see Z)
L Loathing for casinos
M Mountains
N Nightlife
O Oceans (see ; and < too)
P Plantations (see = and 4 too)
Q freQuent/unpredictable/deadly earthQuakes (see 5)
R Rock formations
S ravaged by (also unpredictable) Solar activity (see >)
T Trees
U revolting/Unremarkable dump
V great Volcanoes (see ?)
W evil Water/gargle blasters (both drink and spa)
X eXotic food, also edible poets/moths
Y shYness
Z Zero-G sports (big canyons)
[ loathing of discos  
\ produce exciting sit coms (see 2)
] loathing of sit coms
^ silliness
_ mating traditions
1 unusual Casinos 
2 produce unusual sit coms
3 brew
4 brandy (fields of agriculture)	  
5 occasional Quakes
6 love for tourists (Recreation)
7 love for poetry
8 love for food blenders	
9 loathing of food blenders
: parking meters (for space craft too)
; pink Oceans
< unusual Oceans
= pink Plantations
> occasional Solar activity
? weird Volcanoes
@ Anything is ok (insert into tables for other purposes)

Then we have the planet data:
Each planet has one entry in the table, descriptions can be up to three items.

Code: Select all

char allkey[][8*4] = {
"4S  GT  U   B   6D  C   B   B   ",
"FC  S   X>  D   B   ^D  ^S  I3  ",  
"=Q  O   Z   L>  B   JI  Z   B   ",   
"XA  B   B   GX  3A  OX  B   F   ",   
"?MA ^   U   Y^  B   D   K5  Q   ",   
"U   XC  G   H   GX  C   B   C   ",   
"]C  <9  U   FD  XS  D   Z   X>  ",  
"FA  IW  XO  A   Z3  S   B   XT? ",
"AG  B   XE  U   Y   ]D  AT  B   ", 
"6GX B   F   S   B   LQ  6   NGM ",
"3   D   XO\ XT  ^X  KAM TXQ AQ  ",
"FA  MAC U   IC  B   B   MD  Q   ",
"F   AT  MAG WH  XD  AX  E   S   ",
"8   P   F   Q   LE  B   [G  B   ",
"4GM _N  WAT PG  FGX XAC KM  AC  ",
"^C  M   B   Z4  GM  E   MD  U   ",
"4   D   >   HO  =C  B   TA1 J4  ",
"X^  B   A   ATX VD  D   KM  B   ",
"Q   W>  A   M   U   B   H   D   ",
";Z  MD  Q   ZY  B   XE  MA5 GX  ",  	 
">   ]   7Y  Q   MAC F:  JS  =AX ",
"XD  TAX XO  Q   D   B   AT  B   ",
"AT  <6  U   XS  W>  >   U   AW  ",	
"AS  B   C   MAQ XE  XTD <   B   ",	 
"JS  U   ^   XAD Z   D   Y   D   ",	 
"F   ;1  D   B   J   B   XQ  ?   ",
"NQ  NQ  C   A3  6Q  1O  ZC  Y7  ",	 
"GX  4GT PC  B   TAX KAM W   GX  ",	 
"\]  IS  N   GXT Q   U   Y^  XA  ",	 
"GX  C   XP  A   U   GT  <6  U   ",	 
"B   B   ZW  Q   MQ  GT  B   1D  ",
"B   <   7AT F   GX  AX  B   4   ", /* sys #31 */
"U   Z5  B   B   FN  AX  P   B   ",
"OX  XE  :5  AT  U   >   XE  Q   ",	 
"TAQ H   XMA _   U   HO  L   B   ",	
"1D  YK  Y   C   B   XA? AM5 X4  ",	 
"A   ZS  6AT HF  Q   U   WXT PQ  ",
"XA  WD  NQ  B   GX  NJ  U   AXD ",	 	
"B   FG  Q   B   U   W   NX  C   ",
"6Q  XO  B   ATY K   E   Q   FK  ",	 
"X   MS  GM  XTC C   9   6   1Q  ",
"S   AT  C   =?  H   <A  U   X   ",
"N   U   9   C   Z4  H   XT< I   ",
"C   X^  <   XOC Q   ^C  ATQ >   ",
"Z   U   B   D   GM  D   ^C  <A  ",
"X   B   X   B   XGT B   D   U   ",
"XE  GM  X?  MAS \   TA  I   WQ  ",	 
"AT  9\  Z4  B   N   D   3L  _S  ",	 
"TA] B   ]T  2E  GX  K   8S  GX  ",	
"X   A   U   Q   C   B   XG  4AX ",
"C   Q   IX  8   GX  [A  D   B   ",
"D   JG  TAX XL  F   B   TAQ B   ",
"B   B   XAM PQ  H   AXK C   4   ",
"?   U   D   \M  B   X^  >   3   ",
"_S  AX  WE  C   A   PQ  _:  W   ",
"ZJ  B   TAL B   PZ  U   B   ^W  ",	 
"<MA B   AX4 A5  B   D   Y   E   ",
"E   M   Q   _F  B   \   WC  AT  ",
"B   U   <8  \M  Y   XTD Q   <   ",
"Y   YD  ;\  U   4W  W   Y^  ^C  ",
"3E  ^C  U   U   AT  XS  D   ;S  ",	 
"FGX \   LS  ^   <   AX  OY  XD  ",	
"[A  AQ  MA> AX  ;   D   U   C   ",	 
"YAX 3M  B   NC  B   C   U   _E  ", /* sys #63 */
"P   FA  I7  MQ  C   B   U   IY  ",
"GM  B   XAJ C   ;K  C   XA  FV  ",	 
"D   XD  XO  B   B   4N  XA  2C  ",
"Q   _S  D   GM  S   XTG B   B   ",
"<   1D  H   B   U   2A  L   Z4  ",
"D   B   D   U   Y   XT< Q   B   ",
"FC  XTA NS  M   N   B   B   A   ",	 
"=M  ?   IC  3X  ;5  B   S   6V  ",	 
"Q   B   TA> U   Q   OG  8X  B   ",
"AH  2AT L   A   IC  Y   B   >   ",	 
"<9  XTC B   2>  7Y  P   Y   C   ",	 
"U   U   N4  VG  K   GT  U   B   ",
"J   U   X   A   _   ]C  I   MY  ",
"X   AW  TA  E   ^   =Y  D   E   ",
"Q   K   4   YX  PS  TAS TA] Q   ",
"TMA YX  B   D   3O  U   C   B   ",
"E   _C  2   G   B   GM  >   B   ",
"B   Z   ^   C   VG  >   YAX GX  ",
"B   XC  _   1S  6   N   B   AX  ",
"H   C   B   P2  L   ^X  2   GX  ",
"S   \   7Q  D   YJ  AXJ S   AX  ",
"XTA B   GT  XA  GXT B   M   B   ",
"U   XAD C   B   D   XT  ?   C   ",
"D   GM  A   B   U   \L  J   O   ",
"U   9   Q   TAX IJ  2G  ATS 4GX ",
"B   ^S  U   D   F   GX  B   2S  ",
"D   XM  AME KD  U   ;   D   U   ",
">   NGT _   B   >   A   VG  9   ",
"TAX B   B   U   D   C   \9  I\  ",	
"GM  Q   C   XA  ;D  F   C   PX  ",	 		
"C   B   AS  D   U   1S  6V  ;2  ",
"YZ  I   C   Y   XTP B   W   Q   ", /* sys #95 */	 
"GM  X   U   MQ  ^R  U   U   XAC ",	 
"TAQ B   IZ  L   Z3  \G  I   XA  ",	
"U   Q   B   =MA Y   ]D  4   6XA ",
"U   U   1Q  D   B   \   6F  _C  ",
"4:  I   AZ  U   B   U   C   6   ",	 
"]D  _>  >   4GT A   S   AXV _G  ",	 
"6<  E   B   AD  AT  B   U   Q   ",	 
"B   RAT GM  Q   NJ  ^   U   [E  ",
"YZ  Q   B   ^GT IY  Q   U   XA  ",	
"X   A=  XAT U   ^5  K   2AT GM  ",
"B   B   3   Q   XE  H   C   WA  ",
"U   6?  C   4C  E   G   GX  D   ",	
"\8  Q   WS  XAI \X  Y   Y   AQ  ",	
"]   NQ  _G  W   U   2E  X   A   ",
"F   Y   :5  Y^  M   Q   B   B   ",
"B   FV  GX  AT  ATE AXF U   XK  ",
"L   U   =Q  U   KM  P   3S  JD  ",
"MD  ?   U   S   6AX 6I  \8  E   ",	 
"AZ  N   YS  B   9   :   MAQ XO  ",	    
"K   X   ;Q  N   X   B   B   TA_ ",
"WAT 4C  Q   B   XAI Q   A   MAQ ",	 
"TG  MAS B   Q   U   ATC ATX <E  ",	 
"TAC C   U   B   AT  XOQ A=  WE  ",
"4   AH  B   3   Q   3G  TAM 6   ",
"U   U   U   TA] Q   A   C   IX  ",
"W   D   YD  U   U   MA  E   3   ",
"XA  I   X   A   U   B   B   W   ",	 
"XA  C   U   B   YMA Z   S   B   ",	 
"Y   C   _   ZY  GM  E   Q   Y4  ",
"C   U   ;Q  Y   M>  3   N4  A5  ",
"KV  XD  2>  FA  WD  U   P   _GT ",	 
"=E  Q   U   B   GX  XTK MAS >   ", /* sys #127 */
"3XT >   ^C  V7  B   ^   HN  Q   ",
"B   B   B   XS  Q   P   XAT XP  ",
"B   AMQ _F  O6  ATS I\  <   C   ",
"8S  B   B   C   S   B   ^3  _G  ",
"8S  B   Q   GM  G   U   4   W   ",
"LQ  GT  ^X  >   >   U   MAS Z   ",
"Q   B   AG  C   AW  ^   FX  S   ",
"AGT KM  ATX D   YS  B   E   XAT ",
"JY  D   Q   U   B   GM  B   B   ",
"MAQ ;\  B   FC  M   ZX  F:  C   ",
"X   FI  3;  MQ  ZY  C   XO  N   ",
"XT? Q   U   GXT B   B   ;Q  OX  ",
"MQ  TA2 ID  B   AT  FS  XA  WTA ",
"B   B   9\  XD  N   D   B   A=  ",
"AM  D   U   U   >   A4  LQ  AT1 ",
"FS  A   B   ^GX B   B   XAT A   ",
"B   N3  ATC U   P   B   NX  Q   ",
"_L  \   X   6D  A   ^   XOS AT[ ",
"XA  I   B   D   X   H   U   2>  ",
"PS  GXT >   XA  U   B   U   GX  ",
"R;  C   A   OG  VD  U   B   YZ  ",
"GXT F   XE  _C  Q   4D  ATS O   ",
"TX  PT  Z   A4  B   =W  FC  TAC ",
"F   MQ  NGX _   B   GT  D   ZY  ",
"B   _   _C  AT  Z^  WD  XTG 1   ",
"4V  GX  N   XA  ]   OD  D   XTA ",
"<   6<  N   N   B   U   MAD _3  ",
"6AT TA1 C   U   L_  D   XG  B   ",
"P   D   C   _:  X   B   B   G   ",
"A^  ?   Q   =<  WAT Q   Z^  VE  ",
"B   1D  F   WS  B   C   XA  [G  ",
"W>  KQ  Q   C   7>  W   ^C  B   ", /* sys #159 */
"X   ;Z  MAS Q   F   GXT 4   25  ",
"F   XAT B   6?  9J  1S  B   Q   ",
"8C  Y^  C   2G  WA  B   >   1Q  ",
"B   U   A   8R  VT  XTQ =G  B   ",
"OX  B   MA  >   U   B   A=  FA  ",
"B   KA  XZ  ]XT D   Y   OX  C   ",
"S   B   \L  =D  NX  GM  B   IY  ",
"B   N^  N   XTQ H   XS  YS  Z   ",
"C   JQ  Q   LE  U   OG  3   XTD ",
"D   W   W   C   B   XTC D   B   ",
"XA  LAT 3   D   B   B   A   W5  ",
"4   4>  A   Q   8R  XO\ 4   7^  ",
";Z  Q   AE  4   XD  ^C  W   A   ",
"YS  :   FZ  AN  B   XT< Y   ^W  ",
"Y   AT  B   U   MAD E   8   XY  ",
"VD  FS  C   A   Z   B   U   B   ",
"U   XAZ U   \   6G  B   YJ  MAQ ",
"U   AGT B   NQ  6Q  AT^ D   B   ",
"U   4N  B   <E  Q   D   D   \   ",
"C   B   OX  <   U   B   GT  B   ",
"C   U   D   B   B   GXT Y   F   ",
"4D  B   TX  A   E   B   Q   Y3  ",
"MQ  4I  ]   VY  6Q  ;\  YS  C   ",
"C   X   D   NX  4X  XAT B   ^D  ",
"F   GM  M>  MAD B   C   FZ  B   ",
"XAT 6   M   GX  PGT N>  U   XMA ",
"B   B   7E  [G  XA  U   U   A   ",
"U   B   W:  JS  <   L   AH  B   ",
"AF  B   ZX  6Q  GT  ^W  1Q  U   ",
"4   Z   _G  YX  Y   B   FG  1D  ",
"B   J   ^GX >   J   ^J  FS  MAE ",
"U   B   L   \   U   XT  XA  AT_ ", /* sys #191 */
"^   RA  F   O   B   X   XA  B   ",
"3   WAT A;  B   GX  K2  U   Q   ",
"?MA F   D   C   6<  B   =N  D   ",
"ZS  OT  AT\ XK  A   B   TA; D   ",
"B   Q   WM  K   B   8   Y5  U   ",
"B   4D  AT  P   FS  _   Y7  D   ",
"H   H   4S  6AT FA  B   \   U   ",
"HD  X   U   MAE >   =C  XA  O6  ",
"U   O6  GX  U   B   I   B   B   ",
"X   NGX B   WXT ^?  A   B   ^   ",
"WD  Q   VD  YD  B   ^C  D   B   ",
"Q   OT  NX  GX  XTD U   XOC WE  ",
"_   A   XA  B   B   XG  X   =   ",
"Q   ^   U   AD  Y   ZX  B   B   ",
"WD  6AT ;S  M   X   B   M>  4   ",
"A^  Y   A   B   XI  B   D   P   ",
"B   Q   XY  X   U   X   XGT \   ",
"C   MA  IZ  ZD  NC  2E  ^GX ^   ",
"MA  B   XD  ^R  U   B   GX  _H  ",
"B   NJ  B   _   OG  U   MAT GX  ",
"MAS GX  B   XD  B   AT; Y5  J7  ",
"MD  GXT L   KP  A   I4  ;Q  H   ",
"Q   A3  B   =C  B   Q   U   B   ",
"JP  NJ  AT; Z4  1D  Q   Q   GX  ",
">   N4  U   A;  B   D   L   H:  ",
"J^  YQ  Q   B   :5  U   IC  ^   ",
"F4  B   ^X  MD  JD  A   Q   K5  ",
"B   9\  4   MAQ B   B   XA\ >   ",
"VG  ^C  H   MAS 1Q  U   B   WD  ",
"U   F   Q   GM  D   >   >   HO  ",
"B   M5  B   U   E   HF  D   X^  ",
"MAC ^GT PGX MAD Y   TA  U   N   ", /* sys #223 */
"\8  FGM Q   O   M   U   6D  C   ",
"I   B   GXT Y   AX  =[  6XA <   ",
"C   C   W   U   H:  U   D   B   ",
"MQ  S   JGX AXV AM  W   A=  H   ",
"B   Z4  U   ?   [E  C   D   D   ",
"U   Y   H   B   Q   Y   7Q  XT< ",
"WAT 4GT M>  C   JS  KS  D   AF  ",
":Q  PD  JS  B   B   B   ^   :5  ",
"NX  >   B   YS  KAX XT  TA  P   ",
"E   Y   B   _   B   B   B   =4  ",
"A   ^J  \   E   B   Q   XAT B   ",
"Y   AG  Q   G   B   8C  B   WAT ",
"X   X   ^C  X   B   U   D   D   ",
"Z   B   B   XT  WS  G   JC  C   ",
"B   4C  TAH S   XD  _]  B   Q   ",
"N   W>  FQ  >   Y   \   C   Q   ",
"TA^ A   U   C   B   GM  NW  U   ",
"4   U   B   [G  B   AN  ]XT 4   ",
"AT  B   AT_ N4  M   XTS B   XI  ",
"X>  VY  ATS 9   KAM ;X  GM  AT  ",
"=I  L>  E   AX  C   6I  PT  WS  ",
"D   B   C   _E  A   VE  \[  B   ",
"9Z  S   ATX 2C  ZX  [E  D   6   ",
"TAS E   _   ;K  2   ;S  S   D   ",
"4   Q   D   AH  B   ATQ GT  ^C  ",
"FA  ?   XY  ]   AT  N   XOQ >   ",
"D   =X  GM  X   S   1Q  GT  Z   ",
"XAV XG  _C  MAS TA> TAC NGT P   ",
"MA9 B   Q   ?   C   XS  >   VG  ",
"C   U   U   D   C   S   XD  >   ",
"S   B   AQ  ^GX 4Y  AK  XS  FC  ",
"B   2G  I   XAF O   Q   3   AX  "}; /* sys #255 */
char *key = allkey[isys]+igalaxy*4;  /* base pointer to isys in igalaxy */
int key0 = key[0]-0x20;              /* zero is no key */
Is this the most efficient way to do the job though? I was trying to find alternative approaches.

To reduce space further could the definition be tokenized? Or would that result in a unexceptionable slowdown. Kind of combining how the hobbit rendered text & elites tables. I think with large table structures it might be a good way to prevent excessive size.

Think about this, you got a big list of real life locations and you want to use a method similar to elite to have just the trading/travel element. You walk around using the linked list of a map to all these diffrent locations and trade accordingly.

But this is Avon there are hundreds of places. Ordinary its to much for the spectrum's memory to handle so the data would need to be encoded.

I pick Avon because many of the names of the places are similar so it prevents using methods like ID by the first X characters as you get situations like... Winterbourne & Winterbourne Down. And there are to many similar cases in the data to be able just to do a series of special csae statements.

each location gets its own inventory for trade and a price. That in itself represents a fair amount of data for this many locations. if you have 10 commodities for trade, even if the location has 0 your still looking at 365 x 36500 data points. assuming each could have a maximum amount of 255 thats 8 x 10 bytes per location if its just sitting in memory. 3650 bytes already. Plus the id field for each location. - I am thinking the first 16 chars would be enough for a unique id. so that is 5840 bytes.

What I would like is each time you went to a different location you got a screen with data about the place, it could be done in the same way as elite. But the question is - assuming that the descriptions are fairly limited just assembled in different ways is this the optimum way of doing this?

Assuming there are 365 locations, that would mean the description table would fit quite nicely into the elite method. You wouldn't even need such a large table... Assuming you have a worse case of 3 description strings for each location gives 1095 bytes for the description data table (you don't need an ID field because you assume each location has a static location in the table eg offset 17 from the start of table is Barrow Gurney description.

Code: Select all

Abbots Leigh
Abson
Acton Turville
Aldwick
Almondsbury
Alveston
Arnos Vale
Ashley Down
Ashton Gate
Aust
Avonmouth
Backwell
Bdminton
Bannerdown
Banwell
Baptist Mills
Barrow Gurney
Barrs Court
Barton
Barton Hill
Bath
Bathampton
Batheaston
Bathford
Bathwick
Beach
Bedminster
Bedminster Down
Benter
Binegar
Bishop Sutton
Bishopston
Bishopsworth
Bitton
Blagdon
Bleadon
Bourton
Bradley Stoke
Brentry
Bridgeyate
Brislington
Bristol
Brockley
Burrington
Butcombe
Cameley
Camerton
Chapel Allerton
Carlingcott
Charlcombe
Charmy Down
Charterhouse
Chelvey
Chelwood
Chew Magna
Chew Stoke
Chewton Keynsham
Chewton Mendip
Chilcompton
Chipping Sodbury
Chittening
Churchill
Clandown
Clapton
Clapton in Gordano
Claverham
Claverton
Claverton Down
Clay Hill
Cleeve
Clevedon
Clifton
Clifton Wood
Clutton
Coalpit Heath
Codrington
Coleford
Combe Down
Combe Hay
Compton Dando
Compton Martin
Congresbury
Conham
Coombe Dingle
Corston
Cotham
Cowhill
Dodington
Downend
Doynton
Dundry
Dunkerton
Dunkirk
Durdham Park
Earthcott Green
East Clevedon
East Harptree
East Hewish
East Rolstone
Easter Compton
Eastertown
Easton
Easton in Gordano
Eastville
Edford
Edingworth
Elberton
Elborough
Emborough
Emersons Green
Englishcombe
Failand
Farleigh
Farleigh Hungerford
Farmborough
Farrington Gurney
Faulkland
Felton
Filton
Fishponds
Flax Bourton
Foxcote
Frampton Cotterell
Frenchay
Freshford
Gaunts Earthcott
Greenbank
Grovesend
Gurney Slade
Hallatrow
Hallen
Hambrook
Hamswell
Hanham
Hawkesbury
Hawkesbury Common
Hawkesbury Upton
Haydon
Hemington
Henbury
Henfield
Hengrove
Henleaze
Hewish
High Littleton
Hillesley
Hinton Blewett
Hinton Charterhouse
Holcombe
Horfield
Horton
Hotwells
Hunstrete
Hutton
Inglesbatch
Inglestone Common
Ingst
Iron Acton
Itchington
Iwood
Kelston
Kelston Mills
Kendleshire
Kenn
Kewstoke
Keynsham
Kilmersdon
Kingsdown
Kingston Bridge
Kingston Seymour
Kingswood
Kington
Knap
Knowle
Langford
Langridge
Lansdown
Larkhall
Laverton
Lawrence Hill
Lawrence Weston
Leigh upon Mendip
Leigh Woods
Limpley Stoke
Little Badminton
Little Sodbury
Little Stoke
Littleton
Littleton-upon-Severn
Litton
Locking
Long Ashton
Longwell Green
Lower Claverham
Lower Failand
Lower Hamswell
Lower Hazel
Lower Hounsley
Lower Morton
Lower Shockerwick
Lower Vobster
Lower Weston
Lulsgate
Mangotsfield
Marksbury
Midford
Midsomer Norton
Milton
Monkton Combe
Montpelier
Nailsea
Nailwell
Nempnett Thrubwell
New Cheltenham
Newton
Newton St Loe
North End
North Stoke
North Widcombe
Northend
Northville
Northwick
Norton Hawkfield
Norton Malreward
Norton St Philip
Nye
Oakhill
Old Down
Old Sodbury
Oldbury Naite
Oldbury-on-Severn
Oldland
Oldland Common
Olveston
Over
Patchway
Paulton
Peasedown St John
Pensford
Petty France
Pill
Pilning
Portbury
Portishead
Priston
Publow
Pucklechurch
Pylle Hill
Queen Charlton
Radstock
Rangeworthy
Redcliffe
Redfield
Redhill
Redland
Regil
Rickford
Rowberrow
Rudgeway
Saltford
Sandford
Sea Mills
Severn Beach
Severn Bridge
Shipham
Shirehampton
Shortwood
Shoscombe
Sneyd Park
Snowhill
Soundwell
Southdown
Southmead
Southstoke
Southville
Speedwell
St Agnes
St Andrews
St Annes
St Annes Park
St Augustines
St Catherine
St George
St. Georges, North Somerset
St James, Bristol
St Judes, Bristol
St Pauls
St Philips
St Philips Marsh
St Werburghs
Stanton Drew
Stanton Prior
Stanton Wick
Staple Hill
Stapleton
Star
Stockwood
Stoke Bishop
Stoke Gifford
Stoke St Michael
Ston Easton
Stoney Littleton
Stratton-on-the-Fosse
Swainswick
Swineford
Tadwick
Tellisford
Temple Cloud
Temple Meads
Thornbury
Tickenham
Timsbury
Tockington
Tormarton
Totterdown
Tunley
Twerton
Tyndalls Park
Tyntesfield
Upper Shockerwick
Upper Stanton
Upper Strode
Upper Swainswick
Upper Vobster
Upton Cheyney
Walton Bay
Walton in Gordano
Warleigh
Warmley
Way Wick
Wellow
West Harptree
West Hewish
West Hill
West Town
West Wick
Westbury Park
Westbury-on-Trym
Westerleigh
Weston
Weston in Gordano
Weston-super-Mare
Whitchurch
Whitehall
Whiteway
Wick, Somerset
Wick St Lawrence
Widcombe
Willsbridge
Wilmington
Winford
Winscombe
Winterbourne
Winterbourne Down
Withyditch
Woollard
Woolley
Woolverton
Worle
Worlebury
Wraxall
Wrington
Writhlington
Yate
Yatton
[\code]

Now elite did all this via procedural generation of the names, commodity amounts, positional data. So that required next to no memory. But assuming you were not going to have 2^48 locations.. what's the next best way assuming the limitation of a fixed name list.
Last edited by Nomad on Wed Feb 07, 2018 4:39 am, edited 1 time in total.
User avatar
Seven.FFF
Manic Miner
Posts: 736
Joined: Sat Nov 25, 2017 10:50 pm
Location: USA

Re: How to encode large amounts of text/data.

Post by Seven.FFF »

I’d like to see a sf epic where all the alien races and planets were west country villages.

I don’t think this is suited to your strings, but runtime RLE reencoding is interesting.

http://www.cpcwiki.eu/forum/programming ... -encoding/
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

Seven.FFF wrote: Wed Feb 07, 2018 4:16 am I’d like to see a sf epic where all the alien races and planets were west country villages.
Glad were on the same page :lol: I could make almost legit.. west country villages but its not the same as the actual names if I just go down the elite route of procedural generation. But I could have 2^48 villages going that route. Just they would not be authentic names. (well some of them would be but the majority not so..).

If I just took the unique parts of the names and generated random place names, that would be very efficient place wise - assuming you make another concession. That you can't go back and visit the villages you already have been to. Because of the places being generated on the fly.. its not like the game can remember the previous locations after a certain point. (you cant keep 2^48 locations in memory).

Given those two compromises you could have a massive game. Wouldn't be a good game but a big game none the less. (like an 8-bit no-mans-sky).

If the seed used to generate the rest of the locations was based of a fixed set of values, then shouldn't the generated map then always be the same no matter how many times you play it? a kind of pseudo persistence? Perhaps I am misunderstanding this but if you use like a Fibonacci sequence then apply a modulus to it you should get a repeating sequence (Pisano period) right?

To create the names for the different people is trivial its just +onians/ites/ers Assuming the government types for the location was also linked to that sequence generated it would also be persistent... So even the deals you got between locations would be stable. There would be nothing to stop the game re-rolling the location's stocks levels so that things were a littl' spicy.

hmmm... because how I understand it elite's trading system is just a state machine. All the trades are just dependant on where you are coming from and what type of planet you are trading with is.
User avatar
Seven.FFF
Manic Miner
Posts: 736
Joined: Sat Nov 25, 2017 10:50 pm
Location: USA

Re: How to encode large amounts of text/data.

Post by Seven.FFF »

Yes, I think so. I’ve never looked at Elite code but that how I understood it worked.

I went down a rabbit hole reading about generating fake words with Markov chains, now my brain is fried potato :/
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins
User avatar
Seven.FFF
Manic Miner
Posts: 736
Joined: Sat Nov 25, 2017 10:50 pm
Location: USA

Re: How to encode large amounts of text/data.

Post by Seven.FFF »

I wanted to find out if there were production rules you could use for word generation from the ground up, but people seem to do it by machine learning from the sky down.
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

Seven.FFF wrote: Wed Feb 07, 2018 5:38 am I wanted to find out if there were production rules you could use for word generation from the ground up, but people seem to do it by machine learning from the sky down.
My thought is that seemingly random & independent behaviour could be modelled just by using this technique, when you think about it for the actor and to everyone else around them their actions would appear purely whimsical but would in fact just be following a repeating sequences. This would not be difficult to model either or take that much processing power. Its just a function running off a Pisano period.

That would be a really classical view of a universe though. Still from the games point of view the NPCs are not going to be arguing with you about the merits of the approach. :lol: The important thing was in an 8 bit machine you could do some pretty nice simulations this way.

My idea was just have a jump table of different subroutines, depending on what number you got from the sequence was what that NPC did for that period. This way you would have seemingly varied and random behaviour but it does not suffer the same problem as 'true' random action in NPC because they are still working within a framework that you set. It's a nice way to arrange action groups. You would not need many subroutines for this to look pretty slick. From a processing standpoint, every process is using the same sequence to generate its output so it should in theory be quite efficient.

'Random' events that often slow down the simulations - especially on the 8 bit machines if its just a jump table to a bunch of subroutines based on the sequence. you cut out the major slowdown cause (generating the random input - you are just taking a sequence number and if you want to get fancy apply a trivial manipulation to the value for flavour. But I think its not necessary and for efficiency you could just use it straight. Because I think people forget just how varied this technique can get. Especially when its linked to NPC behaviour.

What is great for testing is that because the seed sequence is constant they always behave in the same way. When you want things to get crazy - then you can add in your modifiers. But I would say avoid doing complicated processing for this phase just use simple math functions to do the job. That way processing is kept manageable and you can use it on more actors in the simulation space.

But yea, I think the top down approach to AI in these kinds of simulations it over-estimates our ability to understand great complexity, when you look at the world around us it would seem to me anyway to be made up of a great many small processes that stack.

Not to get philosophical but I often watched insects, simple animals and noticed how similar there behaviour patterns were. I would imagine at some level each thing would think it was acting in an independent way but in truth it was locked into a pattern they just couldn't get the perspective or had the time to see. When you look at people, at a certain level individual natures seem to not figure in behaviour given enough time.

:lol: That is where I scratch my chin considering how difficult it would really be to get a complex simulation to work I think with the right integer sequences you could do something pretty kick ass even with a 8 bit machine. given enough memory. But with the seed being a constant you can rewind/advance the function to wherever you want to start. there would be no reason to keep large amounts of the sequence in memory. So why not have a number of them running the different processes. You want to get really flash you use the same technique and manipulate what sequence you use. I think 3-4 sequences would give the impression of a very complex simulation space. at least you would have enough seeming divergent inputs to model some cool stuff.

And finally you have to think that given enough time you can see the repetition in the patterns but its an observer related thing. If you can't watch for long enough then it all looks random.

The trick is how efficient could you make the Fibonacci + mod subroutine. as that is the core engine driving the simulation any efficiency saving here impacts the whole model.

space is not going to be a problem because even if you store 1000 data points from the sequence (slough why would you need too?) thats less than 1k of ram...

Thinking about it you probably only need to have 3 values from the sequence in memory. plus your initial seed...

Like even if you stored each number in the fib sequence, by offset 24 from the start of the set you have a number that exceeds 16 bits. (75,025)

So storing the whole sequence in memory 2 byte x 23 gives us 46 bytes to store the whole of the sequence in a lookup table. That removes any processing overhead for the addition. You also have a good idea that your tables will require 23 values to encompass the whole of the sequence data set. Ball park you now already know how many different subroutines you can have in the model with a given memory.

The sequence can't have a value that exceeds 16 bits, so 65536 value the function resets back to the start. Because you know when the sequence is going to reset you can do some pretty cool stuff with the model. Each process you want to be 'different' from the last just use a different mod parameter on the same sequence value. thats the parameter for your lookup table.

so my lookup table for the whole sequence set is

Code: Select all

FIBSEQ:
dw 1,1,2,3
dw 5,8,13,21
dw 34,55,89,144
dw 233,377,610,987
dw 1597,2584,4181,6765
dw 10946,17711,28657,46368 

;or 3 words... 
Example
government actions Mod 7
piracy events Mod 13
markets Mod 5
NPC cycle mod 21

like you might think the mod space is going to be large but when it comes down to it you would only need a handfull of really slick mod functions.

Code: Select all

FUNC8BMOD10:
     LD BC,40000
_LOOP:
     SUB C
     JR NC,$+3
       ADD A,C
     SRL C
     DJNZ _LOOP
     RET
Like that.. (swiped from the z80 heaven site..)

The savings you can make from just forgetting about the 16 bit precision when doing the mod will result into a big time saving in T states. If you just decide to stick to 8 bit mods. Depends if you need the 16 bit precision or not. But I would say for the sake of speed to go for 8 bit.

In this case the bottle neck is going to be the Mod function.

Yes my initial hunch was right, that is an insane amount of T-states even the best mod functions are going to take ages. This is a time that a lookup table is the way to go.. There is no point wasting time having the z80 do the mod on the sequence. Just eat the 46 bytes and store the values for each mod you want.

It would all work off a bunch of jump tables. I think that would be pretty nice way to structure a model.
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

Thinking about this a little bit more over dinner...

The restriction of 16 bits to the Fibonacci sequence only holds true if you actually are going to store the numbers... What we want to do is store the Pisano period.

This means we can encode a much larger sequence that would have been possible if we let the spectrum do the heavy lifting of calculating the Fibonacci sequence and figuring out the Pisano period itself. By directly encoding the Pisano period we can hold many more data points. Thus capturing more repeating sequences.

A much more rich sequence is now possible even if we were to restrict ourselves to 1 byte. (0-255), but this extends a great deal when we go to two bytes.

What is even better is because we are just encoding the sequence until it repeats itself. we can encode the mod of lower value with one byte, then the larger mods with 2 bytes. Giving a good space saving overall.
User avatar
Joefish
Rick Dangerous
Posts: 2042
Joined: Tue Nov 14, 2017 10:26 am

Re: How to encode large amounts of text/data.

Post by Joefish »

Elite uses procedural generation of data. That is, it's all calculated from pseudo-random patterns.

The simplest way of doing it is going to be to take a good 16-bit random number generator routine. This will typically feed the last number they generated back in as the seed for the next number.
If instead you take your co-ordinates on a map as two 8-bit numbers, combine them to get a 16-bit number, and then set that as your seed you can then generate a load of random numbers and use them to describe that location. Each location will be different, but if you return to the same place it will be the same as before. No Man's Sky generates a whole galaxy of planets like this that you can land on and walk around.

The problem with generating place names like this is that if you do it from a list of names, prefixes and suffixes, e.g. Upper / Wotsit / -On-Sea then you're going to get unpredictable repetitions. And if you do it from random syllables or vowel/consonant patterns then you're bound to get the odd offensive one. Maybe better to pick place names by hand and just let the random numbers do the data.

You could do it in advance by having a list of combinations of place names, then run through the map and every place that needs a name picks one from the list; if it's already taken, then it has to pick another. Then you store which combination was assigned to each place. But you have to store the lot there and then. You could do your map in sections, with a different root set of place names for each section, then generate all the place names as you enter that section.

To describe a place, work out its characteristics first like dice-rolling for character stats in a D&D game, then have some code that can chain together sentences to describe them (e.g. pick out particularly good or bad scores to turn into text, then have standby ways of saying everything else is average).

You can generate all the data for a small area and store it, if there are obvious breaks in how you move around. For example, you might just have a town appear as a blob on a world map. When you enter the town, you throw away the world map, generate a town map, and work from that. If you enter a building, you might throw away the town map and generate a building map. Then re-generate the town map when you go back outside.
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

Thanks for your insights joefish, I will go through what you have said and see how it can be applied. The more I think about it the more its totally possible to make a quite substantial procedurally generated game on the spectrum. a bit more thinking on modulating Fibonacci sequences,

with a 64 points of data to play with we get the following modulo.

I have encoded only enough data for the program to generate a repeating sequence. Think of it like a ring. Some modulations are more interesting than other in that they are more varied.

I restricted it to 64 bytes because well I am stingy, if I went crazy and say went to 124 data points then could probably get some more sequences but at the cost of more memory.

Code: Select all

MOD1FIBSEQ:
db 0

MOD2FIBSEQ:
db 1,1,0

MOD3FIBSEQ:
db 1,1,2,0,2,2,1,0

MOD4FIBSEQ:
db 1,1,2,3,1,0

MOD5FIBSEQ:
db 1,1,2,3,0,3,3,1
db 4,0,4,4,3,2,0,2
db 2,4,1,0

MOD6FIBSEQ:
db 1,1,2,3,5,2,1,3
db 4,1,5,0,5,5,4,3
db 1,4,5,3,2,5,1,0 

MOD7FIBSEQ:
db 1,1,2,3,5,1,6,0
db 6,6,5,4,2,6,1,0 

MOD8FIBSEQ:
db 1,1,2,3,5,0,5,5
db 2,7,1,0 

MOD9FIBSEQ:
db 1,1,2,3,5,8,4,3
db 7,1,8,0,8,8,7,6
db 4,1,5,6,2,8,1,0

MOD10FIBSEQ:
db 1,1,2,3,5,8,3,1
db 4,5,9,4,3,7,0,7
db 7,4,1,5,6,1,7,8
db 5,3,8,1,9,0,9,9
db 8,7,5,2,7,9,6,5
db 1,6,7,3,0,3,3,6
db 9,5,4,9,3,2,5,7
db 2,9,1,0

MOD11FIBSEQ:
db  1,1,2,3,5,8,2,10
db  1,0  

MOD12FIBSEQ:
db 1,1,2,3,5,8,1,9
db 10,7,5,0,5,5,10,3
db 1,4,5,9,2,11,1,0 

MOD13FIBSEQ:
db 1,1,2,3,5,8,0,8
db 8,3,11,1,12,0,12,12
db 11,10,8,5,0,5,5,10
db 2,12,1,0

MOD14FIBSEQ:
db 1,1,2,3,5,8,13,7
db 6,13,5,4,9,13,8,7
db 1,8,9,3,12,1,13,0
db 13,13,12,11,9,6,1,7
db 8,1,9,10,5,1,6,7
db 13,6,5,11,2,13,1,0

MOD15FIBSEQ:
db 1,1,2,3,5,8,13,6
db 4,10,14,9,8,2,10,12
db 7,4,11,0,11,11,7,3
db 10,13,8,6,14,5,4,9
db 13,7,5,12,2,14,1,0

MOD16FIBSEQ:
db 1,1,2,3,5,8,13,5
db 2,7,9,0,9,9,2,11
db 13,8,5,13,2,15,1,0
; 24 bytes

MOD17FIBSEQ:
db 1,1,2,3,5,8,13,4
db 0,4,4,8,12,3,15,1
db 16,0,16,16,15,14,12,9
db 4,13,0,13,13,9,5,14
db 2,16,1,0
; 36 bytes

MOD18FIBSEQ:
db 1,1,2,3,5,8,13,3
db 16,1,17,0,17,17,16,15
db 13,10,5,15,2,17,1,0
;24 bytes

MOD19FIBSEQ:
db 1,1,2,3,5,8,13,2
db 15,17,13,11,5,16,2,18
db 1,0 
;18 bytes

MOD20FIBSEQ:
db 1,1,2,3,5,8,13,1
db 14,15,9,4,13,17,10,7
db 17,4,1,5,6,11,17,8
db 5,13,18,11,9,0,9,9
db 18,7,5,12,17,9,6,15
db 1,16,17,13,10,3,13,16
db 9,5,14,19,13,12,5,17
db 2,19,1,0
;60 bytes

MOD21FIBSEQ:
db 1,1,2,3,5,8,13,0
db 13,13,5,18,2,20,1,0
;16 bytes

MOD22FIBSEQ:
db 1,1,2,3,5,8,13,21
db 12,11,1,12,13,3,16,19
db 13,10,1,11,12,1,13,14
db 5,19,2,21,1,0 
;30 bytes

MOD23FIBSEQ:
db 1,1,2,3,5,8,13,21
db 11,9,20,6,3,9,12,21
db 10,8,18,3,21,1,22,0
db 22,22,21,20,18,15,10,2
db 12,14,3,17,20,14,11,2
db 13,15,5,20,2,22,1,0
;48 bytes

MOD24FIBSEQ:
db 1,1,2,3,5,8,13,21
db 10,7,17,0,17,17,10,3
db 13,16,5,21,2,23,1,0
; 24 bytes
[\code]

This already takes up a whopping 604 bytes. But in fairness you get a lot of bang for your buck. No modulation calculations on the sequence necessary and the ability to generate the sequence you need for as long as you want it.

There are some further optimisations that can be made though. The majority of sequences don't require more than 15 digits to encode the sequence. therefore one data point could happy fit into a nibble. so everything before mod 16 could be squished into half the space. (324 / 2) 
So potentially you could squish the whole lot into 442 bytes.

If I went to 128 data points would it really be any more rich.. vs the memory pay off.. 

The thing is a state machine with 24 possible values already is going to have a lot of rich interaction with the player. If I took that to say 48. Given that each location is independent and there is no persistence how much value would this actually add? and would the player notice a move from NPC state machines of 24 possible values to 48? hmmm... interesting.

Luckily someone already has done some back of the napkin calculations regarding modulus of Fibonacci sequences... What is interesting from an encoding standpoint you get fairly regular increase in complexity of the modulus ring but then out of nowhere you get a massive ring that is way larger than its neighbours. If you look at the following link you can see what I mean. the mods to 2000 have been calculated and you can see the complexity of the generated pattern, plus how many zeros you get.  

[url]http://webspace.ship.edu/msrenault/fibonacci/fiblist.htm[/url]

Of course thinking about it, you could store some values so there would be at least some persistent data like faction standing - that wouldn't take a great deal of space to store.  if your faction standing was a maximum of 7 you could encode the faction data into 3 bits so in a worst case say a 30k table could hold faction data for 81920 locations.
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

Luckily someone already has done some back of the napkin calculations regarding modulus of Fibonacci sequences... What is interesting from an encoding standpoint you get fairly regular increase in complexity of the modulus ring but then out of nowhere you get a massive ring that is way larger than its neighbours. If you look at the following link you can see what I mean. the mods to 2000 have been calculated and you can see the complexity of the generated pattern, plus how many zeros you get.

http://webspace.ship.edu/msrenault/fibo ... iblist.htm

Figure its good to share - this website is just fantastic for information about Fibonacci numbers, you can get decades of research on the sequences, theories and related subjects.

https://www.fq.math.ca/books.html

to calculate the modulus of the fibonacci sequence of intrest you can do what I did in R..

Code: Select all

data2 <- c(1,1,2,3,5,8,13,21,34,55,89,144,233,377,610,987,1597,2584,4181,6765,10946,17711,28657,46368,75025,121393,196418,317811,514229,832040,1346269,2178309,3524578,5702887,9227465,14930352,24157817,39088169,63245986,102334155,165580141,267914296,433494437,701408733,1134903170,1836311903,2971215073,4807526976,7778742049,12586269025,20365011074,32951280099,53316291173,86267571272,139583862445,225851433717,365435296162,591286729879,956722026041,1548008755920,2504730781961,4052739537881,6557470319842,10610209857723)

data2 %% 1
data2 %% 2
data2 %% 3
data2 %% 4
data2 %% 5
data2 %% 6
data2 %% 7
data2 %% 8
data2 %% 9
data2 %% 10
data2 %% 11
data2 %% 12
data2 %% 13
data2 %% 14
data2 %% 15
data2 %% 16
data2 %% 17
data2 %% 18
data2 %% 19
data2 %% 20
data2 %% 21
data2 %% 22
data2 %% 23
data2 %% 24 
Then just pick through the output till you find a sequence repeating itself.
Last edited by Nomad on Wed Feb 07, 2018 4:34 pm, edited 3 times in total.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: How to encode large amounts of text/data.

Post by Einar Saukas »

Joefish wrote: Wed Feb 07, 2018 3:37 pmElite uses procedural generation of data. That is, it's all calculated from pseudo-random patterns.
All puzzles in ZEN and ZEN II are procedurally generated. There are 10,000 puzzles in each game and they are all guaranteed to be solvable.

The Sentinel is another good example of procedural generation to produce levels.
User avatar
Joefish
Rick Dangerous
Posts: 2042
Joined: Tue Nov 14, 2017 10:26 am

Re: How to encode large amounts of text/data.

Post by Joefish »

I wouldn't bother with Fibonacci anything. Having it generate unpredictable amounts of data doesn't help.
A pseudo-random number generator is much simpler; you simply ask it for however many numbers you need. If you really want varying amounts of data, use one number to tell you a random amount then ask for that many random numbers.
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

I think part of the problem with the proc-generated games like no-mans sky its a 'thousand miles wide but an inch deep'. Trying to get interesting things to happen and have meaningful interactions I think its the difficult part. But with careful design of the NPC state machines and terain generation/events tables a pretty decent experience could be had.

I remember D&D encounters were ok but if I were to write a game I think I would go more the way of RuneQuest. Much less was left to chance with the rules - the nice thing is much of the rules are just lookup tables (well so is much of D&D) but with RuneQuest a lot of the decisions are taken care of by the tables.

I take your point about just getting a pseudo random number generator that has good characteristics.

What I like about elite is how you can expand the description table generation to do other things like quests. It wouldn't be so difficult to have a table of different quest objectives that could be strung together. Most modern games its just a matter of kill quests, fetch quests, escort quests and location scouting quests. If you built a table that would just build up elements when the player met a quest giver it wouldn't be that difficult to have a great many random quests generated. Much like how star wars galaxies used to handle the missions. Not expecting crazy quality here just enough that it works and keeps people interested.

With the introduction of quests, thats another need to have persistent data - but I think thats a manageable problem. The way I would deal with that is have a limit to the number of active quests/missions and just have a stack type structure. That way you know exactly how much memory to allocate and the player can't go around accepting quests willy nilly and running up a huge amount of memory.

I always felt that for example Darklands could have probably have been implemented as a procedurally generated game and been done a lot faster and within budget. Not to take away from Darklands its a great game but to my mind there was a lot of blood and tears put into that project that could have been saved with a different approach.
User avatar
Seven.FFF
Manic Miner
Posts: 736
Joined: Sat Nov 25, 2017 10:50 pm
Location: USA

Re: How to encode large amounts of text/data.

Post by Seven.FFF »

Nomad wrote: Wed Feb 07, 2018 5:45 pm I think part of the problem with the proc-generated games like no-mans sky its a 'thousand miles wide but an inch deep'.
That is the exact problem. I used to make a lot of generative music on modular synths back in the day, and it's extremely difficult to set up a situation where startlingly unexpected and pleasing things happen, without losing all structure.

I used to describe it as a logarithmic fractal scale of similarity or unexpectedness. Most of your nice random variations and feed-back selected choices have a similarity of 1, where the sweet spot is 3 or 4. Most of the interesting stuff happens where you have a few orders or magnitude of differences, and it's very hard to get in that zone with algorithms.

There's an xkcd about cuils that kinda captures how everything gets too similar after a few levels of abstraction.
Robin Verhagen-Guest
SevenFFF / Threetwosevensixseven / colonel32
NXtel NXTP ESP Update ESP Reset CSpect Plugins
User avatar
Joefish
Rick Dangerous
Posts: 2042
Joined: Tue Nov 14, 2017 10:26 am

Re: How to encode large amounts of text/data.

Post by Joefish »

You're right about games not having so much depth. Although there's been a big update to No Man's Sky that makes it worth exploring. LEGO Worlds uses procedural levels and quests but they do all start to look the same. The problem is that in a purely procedural game, there might be plenty to do, and there might even be grand objectives, but simply knowing that it's random makes you question what the point of it all is.

What I'd suggest would work best is one main plot that's written specifically for the game, and just use the randomness to generate the environment. And maybe some unimportant side-quests. You can record which ones you've done, but it wouldn't kill the atmosphere if you left an area and the game forgot about it, then presented the same quests if you go back. Perhaps a quest generator that just records the types of quest you've done recently, and always offers up something different wherever you go.

A map generator that can interpolate or fill-in around data you fix yourself would be the most useful. You wouldn't use it to generate lots of different games; you make one game that you've tested and are happy with. The randomness just allows you to present a vast world without having to specifically program every little bit of it.

I'd love to be able to understand the 'fractal' code that went into something like Rescue on Fractalus or The Eidolon to generate the scenery. That was adding pseudo-random features to simple structures in real-time. And the features would scale up as you got closer, which is hard to achieve with random data.
Ralf
Rick Dangerous
Posts: 2279
Joined: Mon Nov 13, 2017 11:59 am
Location: Poland

Re: How to encode large amounts of text/data.

Post by Ralf »

When you say auto-generated game I immediately remember Daggerfall, a very famous game from the 90s if you somehow missed it:

http://www.mobygames.com/game/dos/elder ... daggerfall

"Famous" doesn't mean "great" unfortunately. Daggerfall was an incredibly ambitious project which sadly failed in many areas.

It has really huge world but it's mostly randomly generated and it's a weakness. You end with hundreds of towns which look the same as you enter them and thousands of people who also look the same and have names built randomly from some smaller parts. They give you some random quests, somethimes very stupid if you think more.

I remember an example that a game generated a quest to kill some giant spider dwelling in the guild of warriors. You get it, this spider wouldn't have a chance to survive a second in a real guild of warriors ;)

Add hundreds of autogenerated 3D dungeons looking all the same. And tons of bugs, some of them probably coming from the game complexity.

Some people loved it, I didn't like it.

I don't want to play dozens of random but similar levels. I want variety in the game.
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

Einar Saukas wrote: Wed Feb 07, 2018 4:19 pm
All puzzles in ZEN and ZEN II are procedurally generated. There are 10,000 puzzles in each game and they are all guaranteed to be solvable.

The Sentinel is another good example of procedural generation to produce levels.
Thank you for this, being able to see the source code and figure out how the game worked its great.
Seven.FFF wrote: Wed Feb 07, 2018 7:15 pm That is the exact problem. I used to make a lot of generative music on modular synths back in the day, and it's extremely difficult to set up a situation where startlingly unexpected and pleasing things happen, without losing all structure.

I used to describe it as a logarithmic fractal scale of similarity or unexpectedness. Most of your nice random variations and feed-back selected choices have a similarity of 1, where the sweet spot is 3 or 4. Most of the interesting stuff happens where you have a few orders or magnitude of differences, and it's very hard to get in that zone with algorithms.

There's an xkcd about cuils that kinda captures how everything gets too similar after a few levels of abstraction.
Are you a banana plug type of guy or more of the 1/4" jack persuasion? If I had won the lottery I always wanted one of these bad boys. http://www.vintagesynth.com/misc/buchla200.php

:lol:

A lot of generative patches seem to end up in a steady state. But then its better than it sounding like a fart in a wind tunnel after 1 hour of wiggling (Looking at you Daniel Miller.) I think the hardware was around since the 70s its just a matter of applying the right patch. I always was curious what would have happened if a guy like Joseph Schillinger had access to such a machine he would have got a symphony going out of a few patches and a phonebook for input :lol:

But then even using Schillingers routines it has a tendency to sound like lift music. I once spent a month trying to generate drum breaks for a synth project that went nowhere. Something that on paper should be easy has lots of hidden depth. It's like I could generate the breaks but they were so hit and miss. Even with a bunch of example patterns to combine it would always sound kinda off. So I feel your pain about trying to generate a whole track.

With that said, the csound people have done amazing things with that language. I just wish I had the patients to learn to use it properly. That is like having a modular synth inside your terminal :lol: Its weird its not more popular with computer music people but then it does not have a cute gui so I guess that is off putting.

As far as sequences go for the randomness, yea What I like about the modulated Fibonacci sequences is you have some low level variations but also a strong sense or order. Numbers repeat in a predictable way but then out of nowhere you get these spikes of volatility. I think for the game this is a great way to simulate market instability. You have the mods where you get unusual amounts of complexity simulate crashes/rallies.

What gives me the cold sweats is knowing there are sequences and especially in topology - structures that probably would be perfect for the application that we are talking about just gathering dust in a paper or monograph somewhere lol. I think a lot of this stuff exists its just not widely known about.
Joefish wrote: Wed Feb 07, 2018 7:21 pm You're right about games not having so much depth. Although there's been a big update to No Man's Sky that makes it worth exploring. LEGO Worlds uses procedural levels and quests but they do all start to look the same. The problem is that in a purely procedural game, there might be plenty to do, and there might even be grand objectives, but simply knowing that it's random makes you question what the point of it all is.

What I'd suggest would work best is one main plot that's written specifically for the game, and just use the randomness to generate the environment. And maybe some unimportant side-quests. You can record which ones you've done, but it wouldn't kill the atmosphere if you left an area and the game forgot about it, then presented the same quests if you go back. Perhaps a quest generator that just records the types of quest you've done recently, and always offers up something different wherever you go.

A map generator that can interpolate or fill-in around data you fix yourself would be the most useful. You wouldn't use it to generate lots of different games; you make one game that you've tested and are happy with. The randomness just allows you to present a vast world without having to specifically program every little bit of it.

I'd love to be able to understand the 'fractal' code that went into something like Rescue on Fractalus or The Eidolon to generate the scenery. That was adding pseudo-random features to simple structures in real-time. And the features would scale up as you got closer, which is hard to achieve with random data.
There was a really nice documentary made in the early 90s all about fractals. I'll see if I can find it on youtube. Yes fractal geometry is such a rich area for study. But to be honest I don't really understand it enough to be able to use it. I agree with you about having a strong story to be the backbone of the game. It's like generative stuff should be the icing on the cake but not the whole thing. As you say - it should be to enchant an already good/solid game and add depth/extra tasks/behaviours.

One thing I always wondered why it was not done was generating lore, documentation stuff to make a world feel really real. Like you wouldn't need to remember the documents contents after the player had left an area but just generate it on the fly so you could say walk into a study or library and pick up a book and read it. I always found that unrealistic that all the things you could pick up and read were related to you and the quest or interesting important lore. Like in Morowind. There would have been far more documents that were just every day stuff. It would have made it much more of an achievement to find interesting/reliant documents if you had to really search for them. That is a quest in itself. But I kinda understand why you wouldn't want to do that for filthy casual playing the game getting frustrated picking up a invoice or bill for the 5th time in a row. :lol:
Ralf wrote: Wed Feb 07, 2018 7:36 pm When you say auto-generated game I immediately remember Daggerfall, a very famous game from the 90s if you somehow missed it:

http://www.mobygames.com/game/dos/elder ... daggerfall

"Famous" doesn't mean "great" unfortunately. Daggerfall was an incredibly ambitious project which sadly failed in many areas.

It has really huge world but it's mostly randomly generated and it's a weakness. You end with hundreds of towns which look the same as you enter them and thousands of people who also look the same and have names built randomly from some smaller parts. They give you some random quests, somethimes very stupid if you think more.

I remember an example that a game generated a quest to kill some giant spider dwelling in the guild of warriors. You get it, this spider wouldn't have a chance to survive a second in a real guild of warriors ;)

Add hundreds of autogenerated 3D dungeons looking all the same. And tons of bugs, some of them probably coming from the game complexity.

Some people loved it, I didn't like it.

I don't want to play dozens of random but similar levels. I want variety in the game.
I waisted hours in that game :lol: it had its limitations but I still remember forgiving them because it was so much better than anything before. I have to admit I much preferred morowind. It was always a head to head for me between darklands and dagerfall. I liked aspects of both games.

The thing about kill quests generating monsters in unrealistic areas - that seems to happen a lot in the MMOs also. I used to put it down to lazy coding (like how hard is it to have zones in the map where you can't spawn monsters? or just have the npcs react to monsters turning up inside their guild :lol: thinking it through though I can understand not wanting the NPCs to handle the quest for the player. In a large game the slow down would be something else. It would be awesome to watch from a birds eye view multiple NPCs going postal taking the monsters to task all over the game world. The player would be walking through a aftermath. Thinking about it thats exactly what you want to happen in a first person 'war simulator' type game.
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

As a quick addition, as we were talking about music and generating notes.

Pisano points generated from the Modulo 7 (or if you are into early music or alternatively avant garde another number base)..

Map each generated integer value to a note in your scale.

A - 0
B - 1
C - 2
D - 3
E - 4
F - 5
G - 6

Given the sequence you then get

B B C D F B G A G G F E C G B A

But wait there is more, you can use the same technique to obtain your note lengths, the tempo, the octave..

Lol heres my little ditty with it in Beepola..

https://ufile.io/gu3re

If you were to do this with sample based music you can just assign the integer value to your sample.

Hi Hat Open - 0
Hi Hat Closed - 1
Snare - 2
Kick - 3
Base - 4
Tom - 5
Cowbell - 6

To make it a bit more interesting - map a different modulo sequence and use that to indicate what pattern to play in the song sequence...

It's a trivial method but if you were pressed for time and wanted to try a bunch of patterns that is how I would do it. You are bound to find something that would be ok..

Stockhausen's lectures are good for thinking about how to write music using algorithms.

[youtube]https://youtu.be/lYmMXB0e17E[/youtube]

There is a whole series you can find them with that. Stockhausen is a bit Marmite; I understand but even if you can't stand his music or as a person I think the method is sound.
User avatar
R-Tape
Site Admin
Posts: 6353
Joined: Thu Nov 09, 2017 11:46 am

Re: How to encode large amounts of text/data.

Post by R-Tape »

Lol heres my little ditty with it in Beepola..
Mostly painless.
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

R-Tape wrote: Thu Feb 08, 2018 9:14 am
Lol heres my little ditty with it in Beepola..
Mostly painless.
I'll get me coat :lol: structure of the 'tune' ideally would have gotten a lot more love. the main channel, second channel and drums are all using that sequence. (Lol just to show it could be done if it was 'legit' I would have used a more conventional drum track.) The mod5 pattern might be good to spice things up a bit. I would have to have a play.

Could just use the shillinger method and pick a phone number, That works just as well. :lol:

In fairness the rest of the mods produce some interesting dittys also. combining the various patterns into a complete song. I am crap at composition haha I will be honest I would struggle to beat some of these generated melody's they sound ok. Especially if you are up against it time wise these cheats seem to help.

Makes you wonder what the other modulations sound like I only tested it up to 8.. Could get really creative and move away from equal temperament .

Another simple hack, turn the sequence into a set. (thereby removing all duplicate notes in the sequence).

Another hack. take the set and now sprinkle in the missing notes from the scale (if any).

:lol: There are loads of things you can do - this will result in starting point for composition. Like unless you are like super self disciplined I think everyone cheats with these and wiggles the sequences to make them sound better. That was the joke about Serial music nobody followed the rules anyway.
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: How to encode large amounts of text/data.

Post by Nomad »

Seven.FFF wrote: Wed Feb 07, 2018 4:16 am I’d like to see a sf epic where all the alien races and planets were west country villages.

I don’t think this is suited to your strings, but runtime RLE reencoding is interesting.

http://www.cpcwiki.eu/forum/programming ... -encoding/
Not quite sure how I missed this. :lol: hmmm well for tables where there is lots of continuous repeating data bytes this would be all kinds of awesome. So if you had like a sparse array with tons of 0's this would be really sweet.

Code: Select all

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
now becomes*

Code: Select all

00177 0100 00249 0100 0068 0100 00176 0100 0093
*I think...
Wall_Axe
Manic Miner
Posts: 492
Joined: Mon Nov 13, 2017 11:13 pm

Re: How to encode large amounts of text/data.

Post by Wall_Axe »

so he is saying that in the game north star the 'screen decoration' ie the still graphics around the gameplay area are unpacked onto the screen?

I take it you would have to unpack it to read a certain set of coords?
Post Reply