New WoS and ZXDB

This is the place for general discussion and updates about the ZXDB Database. This forum is not specific to Spectrum Computing.

Moderator: druellan

Post Reply
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

New WoS and ZXDB

Post by Einar Saukas »

It turns out new WoS is using ZXDB! More precisely ZXDB version 1.08 released on September 2, 2018.

Don't you believe me? Then visit new WoS software page and click on "EXPORT CSV (ALL)", to download some of the data from all titles stored in new WoS. Afterwards see if you can find any differences between new WoS content and the ZXDB version I mentioned... Or visit a few game pages at random to compare old WoS (search for a game in Google then click "cached"), new WoS, and any other site running ZXDB like SpectrumComputing or ZXInfo.

Don't take the wrong, I'm glad that new WoS is using ZXDB. Rebuilding WoS was exactly the reason ZXDB was created. I have been trying for years to convince Lee Fogarty that it would be a terrible idea otherwise (as recently as last week). The community doesn't deserve to have everyone's contributions thrown away, restarting everything from scratch for absolutely no reason. Besides, ZXDB is an open database, everyone is welcome to use it, there's no need to even ask my permission.

So what's the problem?

The problem is using ZXDB while pretending otherwise. Last month (May 2020), Lee Fogarty explicitly claimed new WoS would initially use "old WoS data" (although WoS hasn't been updated since 2013) and they were "going through the data applying fixes", fixing "thousands and thousands of errors", etc. However this claim is false: new Wos is not using either. It's using ZXDB from 2018. So now you know where thousands and thousands of fixes came from.

Even worse, there was clearly an attempt to disguise using ZXDB. For instance, ZXDB version 1.08 contained 24369 non-Timex programs from old WoS (from ID=1 to ID=28187), 6011 programs added in ZXDB (from ID=28201 to ID=34414), and 137 Timex programs from old WoS (from ID=4000001 to ID=4000143). New WoS is using all data from ZXDB 2018, but all IDs above 28187 were removed, so it only shows the same 24369 titles that already existed in old WoS. Well, actually new WoS indicates 24368 titles because somehow they lost Trans-Disc Express along the way. Therefore you currently won't find any titles in new WoS that were added after 2013... although it still contain several references to them! A few examples:

* Comercio Cosmico was released in 1987. Later I re-released it myself in English as Cosmic Commerce in 2015. Old WoS from 2013 didn't know it, but this re-release appears at new WoS (click on "RELEASES" to see it).

* En Busca de Mortadelo was re-released by Team Siglo XXI in 2017. Again old WoS didn't know it, but new WoS does.

* Catacombs of Balachor was a single game in old WoS. After a sequel was released in 2014, this game was added to series "Balachor" in ZXDB. You won't find the sequel at new WoS, but Catacombs of Balachor at new WoS lists series "Balachor", that only contains "Catacombs of Balachor". Isn't it weird to see a series of only one game?

* Alter Ego was a single game in old WoS. After a sequel was released in 2014, this game was added to series "Alter Ego" in ZXDB. You won't find the sequel at new WoS, but Alter Ego at new WoS lists a series called "Alter Ego", that only contains game "Alter Ego".

There are other ways to identify this data came from ZXDB. For instance, old WoS didn't distinguish text adventures, therefore Colossal Adventure and The Hobbit were both classified as "Adventure: Text". In ZXDB 2018, we added the distinction between "Adventure Game: Text-Only" (like Colossal Adventure) and "Adventure Game: Text/Illustrated" (like The Hobbit). Coincidentally that's what you see at new WoS pages for Colossal Adventure and The Hobbit.

But wait, last week I insisted once again that new WoS should use ZXDB. So perhaps my argument finally convinced Lee Fogarty to change his mind, but then he "forgot" to mention it? Not really. I wrote it last week (June 2020), when I was accused of "stealing" WoS data (proved wrong), copying another database and not crediting other people properly for their participation on creating it (also proved wrong). But if you visit any game page at new WoS at random, you will notice almost all of them are dated May 2020. Which means they falsely accused me of doing it, while they were already doing themselves exactly this! Incredible.

But wait, perhaps ZXDB 2018 was not copied directly, these are just coincidences? I mean, it's theoretically possible that every single change made in ZXDB for the first 2 years was added independently to the old WoS database. After all, most of these changes came from posts publicly available in discussion forums. Or perhaps someone simply saw information in other sites using ZXDB, then typed them again into old WoS... Right?

Wrong. New WOS is a straight copy directly from ZXDB 2018, period. How do we know? Well, let's dive a little deeper.

Martijn's WoS internal file maindb.dat contained the following comment for BIFROST* ENGINE:

Code: Select all

Inspired by {ZXodus Engine|Andrew Owen [2]}.
Based on it, old WoS produced the following HTML:

Code: Select all

Inspired by <a href="http://www.worldofspectrum.org/infoseek.cgi?regexp=^ZXodus+Engine$&pub=^Andrew+Owen+[2]$&loadpics=1">ZXodus Engine</a>.
By comparison, ZXDB database file contains the following comment for the same title:

Code: Select all

Multicolor 8x1 Graphics Engine (18 char columns x 18 char rows). Inspired by {ZXodus Engine|Andrew Owen [2]|0026639}.
Although this is not what you see at websites, because each site uses this information from ZXDB to generate their own HTML. This is actually what you see at SpectrumComputing:

Code: Select all

Multicolor 8x1 Graphics Engine (18 char columns x 18 char rows). Inspired by  <a href="index.php?cat=96&id=0026639">ZXodus Engine</a>.
This is what you see at ZXInfo:

Code: Select all

Multicolor 8x1 Graphics Engine (18 char columns x 18 char rows). Inspired by <a href="https://zxinfo.dk/details/0026639">ZXodus Engine</a> (Andrew Owen [2]).
And this is what you see at new WoS site:

Code: Select all

Multicolor 8x1 Graphics Engine (18 char columns x 18 char rows). Inspired by {ZXodus Engine|Andrew Owen [2]|0026639}.
It means this line wasn't even copied from another site. It was imported straight from ZXDB source file. Just like everything else.

Also according to new WoS, BIFROST* belongs to a group called "BIFROST/NIRVANA" (exactly like in ZXDB), although NIRVANA itself was removed from new WoS after importing ZXDB.

Another interesting point is, there are 2 Spectrum authors called "Andrew Owen", so this Andrew was originally added to old WoS as "Andrew Owen [2]". Years ago Andrew told me by email that his middle initial was "S", so I changed ZXDB from "Andrew Owen [2]" to "Andrew S. Owen". However I forgot to update this comment... Coincidentally new WoS made the same mistake! Thus leading to our next example:

I co-authored "FZX" with Andrew in 2013. He was originally credited as "Andrew Owen [2]" in old WoS. He's now credited as "Andrew S. Owen" in ZXDB. But if you look at new WoS, you will notice he's credited both ways: author "Andrew S. Owen" and publisher "Andrew Owen [2]" (open "RELEASES" below to see it). So what's the reason for this inconsistency? Because the list of publishers in new WoS came mostly from Martijn's WoS internal file publishers.dat, that Lee Fogarty imported back in May 2017 and never fixed, but all actual game data came from ZXDB. How do we know? Because game data at new WoS contains all information that was updated in ZXDB 2018, but the list of publishers at new WoS still contains all the bugs I reported when I started ZXDB in 2016. For instance Waner Brothers was reported in July 2016.

OK, so I made it clear that new WoS is using an old version of ZXDB. But wait, these people also helped me to import old WoS content into ZXDB at the beginning. Therefore we should excuse them for using the result of their own work without crediting ZXDB, right? The problem is, I imported old WoS content with their help from July 2016 to August 2016. If they had used one of the early versions of ZXDB that they participated, without crediting ZXDB, I would leave it alone. However they chose to use a version of ZXDB from September 2018, in order to take advantage of over 2 years of other people's work, without crediting anybody. That's a ZXDB version released 2 years after WoS stopped supporting ZXDB and started attacking my work. About 1 year after ZXDB and SpectrumComputing were censored at WoS thus forcing ZXDB to move to another forum. Months after I was personally censored at WoS without ever receiving any explanation.

This leads to our next question. Here's a list of ZXDB releases since we started numbering them in early 2018. Why did they chose to use ZXDB version 1.08 from September 2018? It cannot be that they got the most recent version available at the time, because if they took it before ZXDB 1.09 (October 2018), it would mean they have been lying for the last 2 years about importing data from old WoS and working on thousands of fixes themselves. It cannot be that they only started using ZXDB recently but deliberately downloaded an older version of ZXDB to make it harder for people to notice, because that would be really dishonest...

The bottom line is, it's now clear WoS made little progress for years, despite claiming they were busy working on a much better database containing thousands and thousands of fixes. In the meantime, they initially made excuses for not using ZXDB, then attacked our work with false accusations. Half a decade later, when a new WoS is finally launched, it's simply using ZXDB in disguise. And it happened only 4 days after they falsely accused ZXDB of copying another database without crediting people's work! How ironic.

What now? Should new WoS stop using ZXDB? Of course not! To stop doing it now, after getting caught, would only make it look even worse. A new site using even older data, full of errors already fixed for the last few years, won't help anybody in the community. Instead, I would simply expect an apology for this stunt, and to properly credit somewhere in new WoS that it's using content originally from ZXDB (according to the ZXDB license). Easy, right?

Also for heaven's sake, use an updated version of ZXDB! Why take 2 years of other people's work while trying to avoid crediting them, when you could have taken all 4 years!!!


NOTE: I have just posted this text above at WoS forum, then I thought this subject would be of interest here too...

EDIT: Just fixed a missing link.
User avatar
Juan F. Ramirez
Bugaboo
Posts: 5102
Joined: Tue Nov 14, 2017 6:55 am
Location: Málaga, Spain

Re: New WoS and ZXDB

Post by Juan F. Ramirez »

Einar Saukas wrote: Thu Jun 18, 2020 12:03 pm It turns out new WoS is using ZXDB!
I can't believe it! :D
User avatar
Pegaz
Dynamite Dan
Posts: 1209
Joined: Mon Nov 13, 2017 1:44 pm

Re: New WoS and ZXDB

Post by Pegaz »

Einar Saukas wrote: Thu Jun 18, 2020 12:03 pm
Also for heaven's sake, use an updated version of ZXDB! Why take 2 years of other people's work while trying to avoid crediting them, when you could have taken all 4 years!!!
Why?
We all know why, Einar and I will not repeat again what I have written many times on similar occasions.
Maybe this way, this topic will avoid the fate of being locked.
If you still have some illusions or you haven't understood who you are dealing with, then it is pointless to say anything else.
It was enough to look at the reaction to your recent messages there - just total ignorance.
Real life is not a fairy tale and good does not always defeat evil, it's usually the opposite.
I really appreciate your work and effort and I would not like to give advices.
Still, maybe it's time to remember the good old Bible saying - Don't throw your pearls before the pigs...
User avatar
StooB
Dynamite Dan
Posts: 1073
Joined: Mon Oct 22, 2018 11:03 am
Contact:

Re: New WoS and ZXDB

Post by StooB »

I think it's more likely to be using data taken from ZXDB at that point of time, rather than actually running from an instance of ZXDB itself.
User avatar
bob_fossil
Manic Miner
Posts: 654
Joined: Mon Nov 13, 2017 6:09 pm

Re: New WoS and ZXDB

Post by bob_fossil »

Einar, any chance you can share the POKES you're using to give yourself infinite patience and infinite reasonability? Thanks. :)
User avatar
PeterJ
Site Admin
Posts: 6855
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New WoS and ZXDB

Post by PeterJ »

Maybe [mention]StooB[/mention]. Regardless, there is still a requirement in the open source license to mention ZXDB, and a request to make the created database (of one was created) open too.

It's obviously good though that it's being used. Although to have rubbished all the work of Einar and others previously (even up to last week) and make numerous false claims, then not mention it turns out ZXDB has been used is disgraceful.
User avatar
Pegaz
Dynamite Dan
Posts: 1209
Joined: Mon Nov 13, 2017 1:44 pm

Re: New WoS and ZXDB

Post by Pegaz »

StooB wrote: Thu Jun 18, 2020 12:43 pm I think it's more likely to be using data taken from ZXDB at that point of time, rather than actually running from an instance of ZXDB itself.
Well, Einar gave quite detailed arguments for his claims.
We will see soon what the response will be and who was right...
User avatar
Rorthron
Dynamite Dan
Posts: 1644
Joined: Sun Nov 12, 2017 10:35 pm

Re: New WoS and ZXDB

Post by Rorthron »

Juan F. Ramirez wrote: Thu Jun 18, 2020 12:35 pm
Einar Saukas wrote: Thu Jun 18, 2020 12:03 pm It turns out new WoS is using ZXDB!
I can't believe it! :D
I can, and am not in the least surprised. I have long suspected Lee would pull such a stunt. There seemed to be no other way to catch up with the massive backlog of changes the WoS archive needed.
User avatar
Rorthron
Dynamite Dan
Posts: 1644
Joined: Sun Nov 12, 2017 10:35 pm

Re: New WoS and ZXDB

Post by Rorthron »

Pegaz wrote: Thu Jun 18, 2020 1:01 pm Well, Einar gave quite detailed arguments for his claims.
We will see soon what the response will be and who was right...
What are the odds that Lee will just ignore Einar's arguments and respond with some feeble and false whataboutery?
User avatar
StooB
Dynamite Dan
Posts: 1073
Joined: Mon Oct 22, 2018 11:03 am
Contact:

Re: New WoS and ZXDB

Post by StooB »

Pegaz wrote: Thu Jun 18, 2020 1:01 pm
StooB wrote: Thu Jun 18, 2020 12:43 pm I think it's more likely to be using data taken from ZXDB at that point of time, rather than actually running from an instance of ZXDB itself.
Well, Einar gave quite detailed arguments for his claims.
We will see soon what the response will be and who was right...
I'm not saying Einar's wrong! I'm saying that Lee Fogarty would rather take the data and put it in his own database because he doesn't like the fact he didn't design ZXDB, despite it making any updates into a huge data conversion mess. Which means WoS is now broken and frozen in time in 2018, rather than broken and frozen in time in 2014.
User avatar
cha05e90
Dizzy
Posts: 66
Joined: Sun Nov 19, 2017 7:02 pm

Re: New WoS and ZXDB

Post by cha05e90 »

StooB wrote: Thu Jun 18, 2020 1:18 pm Which means WoS is now broken and frozen in time in 2018, rather than broken and frozen in time in 2014.
Amen.
48K/+/+/+/+128K/Vega/Next&80/81&88
User avatar
Pegaz
Dynamite Dan
Posts: 1209
Joined: Mon Nov 13, 2017 1:44 pm

Re: New WoS and ZXDB

Post by Pegaz »

StooB wrote: Thu Jun 18, 2020 1:18 pm
Pegaz wrote: Thu Jun 18, 2020 1:01 pm Well, Einar gave quite detailed arguments for his claims.
We will see soon what the response will be and who was right...
I'm not saying Einar's wrong! I'm saying that Lee Fogarty would rather take the data and put it in his own database because he doesn't like the fact he didn't design ZXDB, despite it making any updates into a huge data conversion mess. Which means WoS is now broken and frozen in time in 2018, rather than broken and frozen in time in 2014.
Yes, but as far as I understood explanation from Einar initial post, he not only took the data from 2018, but the complete ZXDB structure.
User avatar
moroz1999
Manic Miner
Posts: 329
Joined: Fri Mar 30, 2018 9:22 pm

Re: New WoS and ZXDB

Post by moroz1999 »

That's just a hilarous finding, I can't stop laughing!
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New WoS and ZXDB

Post by Einar Saukas »

The new WoS homepage has just changed. About 1 hour ago, immediately after I posted, I took a screenshot and it was indicating 24368 titles:

Image

Now I checked again and this information disappeared:

Image

Although it now says it was last updated 21 hours ago. Weird!
User avatar
moroz1999
Manic Miner
Posts: 329
Joined: Fri Mar 30, 2018 9:22 pm

Re: New WoS and ZXDB

Post by moroz1999 »

Pegaz wrote: Thu Jun 18, 2020 1:29 pm Yes, but as far as I understood explanation from Einar initial post, he not only took the data from 2018, but the complete ZXDB structure.
It seems to me that they have made some kind of information import, similar to what I do in ZX-Art. They have provided the same possibilities in their database for information storing and made some kind of migration script.
https://worldofspectrum.org/infoseekid. ... loadpics=1 - here for example David Whittaker is mentioned two times. It looks like this contains both old information (from WoS?) and a new one from ZXDB. This wouldn't occur if they used ZXDB straightly.

Image
User avatar
PeterJ
Site Admin
Posts: 6855
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New WoS and ZXDB

Post by PeterJ »

I think that maybe is due to him not understanding how the roles tables work [mention]moroz1999[/mention]
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New WoS and ZXDB

Post by Einar Saukas »

moroz1999 wrote: Thu Jun 18, 2020 1:43 pmhttps://worldofspectrum.org/infoseekid. ... loadpics=1 - here for example David Whittaker is mentioned two times. It looks like this contains both old information (from WoS?) and a new one from ZXDB. This wouldn't occur if they used ZXDB straightly.

Image
No, that's not it.

They are using ZXDB without understanding how it works. More specifically how to combine ZXDB tables "authors" and "roles".

Each author of a certain software is stored once in ZXDB table "authors", and stored multiple times in ZXDB table "roles" (once for each role). However new WoS is showing each author once from table "authors" (with "Unknown" role), then again for each time it appears in table "roles".

For instance, ZXDB didn't store roles for Denis and Shiru, so they are only stored in "authors":

https://worldofspectrum.org/archive/sof ... retrosouls

ZXDB only had 1 role for author Olli Hihnalla, so he's stored once in "authors" and again in "roles":

https://worldofspectrum.org/archive/sof ... sasoft-ltd

And ZXDB stored multiple roles for these authors, so each one is stored once in "authors' and multiple times in "roles":

https://worldofspectrum.org/archive/sof ... nar-saukas

That's the problem of using other people's work without knowing what you are doing.
User avatar
moroz1999
Manic Miner
Posts: 329
Joined: Fri Mar 30, 2018 9:22 pm

Re: New WoS and ZXDB

Post by moroz1999 »

Then this all is even more crazy than sounded in the beginning :)
User avatar
PeterJ
Site Admin
Posts: 6855
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New WoS and ZXDB

Post by PeterJ »

Obviously, [mention]fogartylee[/mention] is welcome to join this discussion and explain any misconceptions.
User avatar
moroz1999
Manic Miner
Posts: 329
Joined: Fri Mar 30, 2018 9:22 pm

Re: New WoS and ZXDB

Post by moroz1999 »

https://worldofspectrum.org/archive/pub ... /moroz1999 - they display me from Russia, but this was long ago corrected to Estonia in ZXDB.
Also a wrong label Death Moroz is credited to me (also fixed in ZXDB a long time ago).

And I cannot find our Crystal Kingdom Dizzy 2017 in WoS database for some reason.
User avatar
PeterJ
Site Admin
Posts: 6855
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New WoS and ZXDB

Post by PeterJ »

moroz1999 wrote: Thu Jun 18, 2020 2:23 pm And I cannot find our Crystal Kingdom Dizzy 2017 in WoS database for some reason.
Any titles added with new infoseek IDs have been removed.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New WoS and ZXDB

Post by Einar Saukas »

moroz1999 wrote: Thu Jun 18, 2020 2:23 pm https://worldofspectrum.org/archive/pub ... /moroz1999 - they display me from Russia, but this was long ago corrected to Estonia in ZXDB.
Also a wrong label Death Moroz is credited to me (also fixed in ZXDB a long time ago).
Because the list of publishers in new WoS came mostly from Martijn's WoS internal file publishers.dat, that Lee Fogarty imported back in May 2017 and never fixed. Although the rest of the game data came from ZXDB.

moroz1999 wrote: Thu Jun 18, 2020 2:23 pmAnd I cannot find our Crystal Kingdom Dizzy 2017 in WoS database for some reason.
Because all titles with IDs above 28187 (the last ID added to old WoS) were removed, in an obvious attempt to disguise that new WoS is using ZXDB.
Ralf
Rick Dangerous
Posts: 2279
Joined: Mon Nov 13, 2017 11:59 am
Location: Poland

Re: New WoS and ZXDB

Post by Ralf »

Hahahahaha :lol:
I guess you shoud take it as a compliment and some proof that you were right, Einar.

Actually I expected something like that. That they will be too lazy to search on internet for all titles released after 2013 and will eventually use data from ZXDB.

But i didn't expect that they will turn stealth mode on and will use ZXDB pretending that they are not using it :lol: But will do it in a lame way so it will be obvious that they are using it. And they will cut out all new data.

And we have now a new WOS. which runs slower and lacks a lot of content present on old version but has some fancy animations instead.

Well, I'm really happy that we have Spectrum Computing :)
catmeows
Manic Miner
Posts: 711
Joined: Tue May 28, 2019 12:02 pm
Location: Prague

Re: New WoS and ZXDB

Post by catmeows »

While I understand amusement and satisfaction coming from the fact WOS is using ZXDB, I'm little bit sad because of hostility developing between WOS and SC :shock:
Proud owner of Didaktik M
User avatar
moroz1999
Manic Miner
Posts: 329
Joined: Fri Mar 30, 2018 9:22 pm

Re: New WoS and ZXDB

Post by moroz1999 »

Einar Saukas wrote: Thu Jun 18, 2020 2:39 pm Because the list of publishers in new WoS came mostly from Martijn's WoS internal file publishers.dat, that Lee Fogarty imported back in May 2017 and never fixed. Although the rest of the game data came from ZXDB.

Because all titles with IDs above 28187 (the last ID added to old WoS) were removed, in an obvious attempt to disguise that new WoS is using ZXDB.
Thank you! Sounds really like true.
Post Reply