New WoS and ZXDB

[Pobulous]

No-one has posted this, yet, from the WOS facebook group: This was buried in a thread asking why WOS error pages were leaking browser and location data from other users.

Lee Fogarty
I think it needs clearing up exactly what "created by" means. And another reason the data has to go is some of the changes are dubious or un-needed. This is why WoS always wanted to credit people submitting changes, and list them on the whats new page.
From what I can make out, the claim is that we are using the ZXDB database. Totally untrue - the WoS db was created a long time before - using the original data files.
Any group of people creating a database from an existing dataset will invariably create similar tables and structure. Things such as the machine types used - create a list for machine types.. both parties will likely create the same table with the same data. There is a WoS admins group on FB that Einar was in, and posts still there where I am sending structure/data to him.
That seems to be changing now to we are using "their" data. Again - untrue. There are some left over bits from a very old import test that are being removed.
WoS currently has over 300,000 indexed pages. Not just software. The software is a very small part of the database, and with the bits we are removing, comes to a minuscule amount.
This is all something that could have been sorted with a PM.

If it's simple to resolve via a PM, why has Lee not resolved it via a PM?

[ketmar]

and not only with the same data, but with the same comments that wasn't present in the original db, and even with the same infoseek ids for things that weren't in the original db! this is a pure coincidence, of course.

[Mike Davies]

Lee Fogarty
I think it needs clearing up exactly what "created by" means. And another reason the data has to go is some of the changes are dubious or un-needed. This is why WoS always wanted to credit people submitting changes, and list them on the whats new page.
From what I can make out, the claim is that we are using the ZXDB database. Totally untrue - the WoS db was created a long time before - using the original data files.
Any group of people creating a database from an existing dataset will invariably create similar tables and structure. Things such as the machine types used - create a list for machine types.. both parties will likely create the same table with the same data. There is a WoS admins group on FB that Einar was in, and posts still there where I am sending structure/data to him.
That seems to be changing now to we are using "their" data. Again - untrue. There are some left over bits from a very old import test that are being removed.
WoS currently has over 300,000 indexed pages. Not just software. The software is a very small part of the database, and with the bits we are removing, comes to a minuscule amount.
This is all something that could have been sorted with a PM.

And thus begins the attempt to gaslight people. Notice he doesn't cover any of the substance in Einar's statements. Just brush it off. The data examples Einar has shown did not exist in the WoS datafiles ZXDB was founded on, they were added into ZXDB -- mainly from Einar himself! -- and those changes aren't those where one person suggests a correction into two sources - it's clear that the source of the current WoS data is from ZXDB.

This is not good enough, Fogarty. It doesn't explain how text that is bespoke to ZXDB appears in WoS. It doesn't explain how image references that are specific to ZXDB are appearing in WoS. Not by a country mile is this explanation is an explanation. It's just another attempt to gaslight the community. If he's serious about wanting to resolve this, he needs to stop with the gaslighting, and enable an ability to make direct comments to direct statements that Einar has published.

Again, this is how this goes:

* Fogarty and Chandler issue a public apology to Einar that meets with Einar's satisfaction
* WoS states clearly that its updated data is from ZXDB, and it will abide by the licensing conditions of it.

Notice Fogarty is admitting to deleting data when it's been reported -- why only one issue at a time? This is called list-washing, the intention is to minimise the visibly breach of licensing, while still violating the licenseing terms. Fogarty should delete ALL DATA that came from ZXDB, regardless if it's reported or not, if his intention is for WoS to not use ZXDB.

It's a simple case of rolling back all the data to the last old-WoS update, and applying only the changes that didn't come from ZXDB.

Fogarty: This is all something that could have been sorted with a PM.

And in the time between 18 June 2020 through to 2 July, why hasn't Fogarty responded publicly or sorted with a PM Einar's concerns. That's two weeks, most of it silent where he could have talked this out with Einar. Why has he failed to explain why ZXDB-specific data appears in WoS pages? And how does that happen without touching ZXDB data?

[kolbeck]

Why has he failed to explain why ZXDB-specific data appears in WoS pages? And how does that happen without touching ZXDB data?

If you ask kids the same type of question, the answer is “by magic”

[PeterJ]

It doesn't explain how text that is bespoke to ZXDB appears in WoS. It doesn't explain how image references that are specific to ZXDB are appearing in WoS.

Very good points [mention]Mike Davies[/mention].

[moroz1999]

My thoughts:
There wont be any peaceful resolution.
Lee won't admit that he has used a lot of ZXDB information.
No apologies would be made.
Current WOS team is lost to common sense or logical actions.

We need to accept this and move on, that's the new reality we are living in:
1. There won't be one single ZX information archive anymore. There was never, and this illusion will be lost completely.
2. Prepare for some actions from WOS. I cant tell what to wait for (DDOS, legal queries, gaslighting in social media - whatever), but don't let them take you by surprise. Search for the weak points and start thinking about defense and possible strategies in different situations.
3. The WOS won't be the only problem. As the community would grow, you would have to take measures you wont like now. It's better to think at about it now and formulate different situations for yourselves now already.

[RWAC]

Lee won't admit that he has used a lot of ZXDB information.

Despite evidence to the contrary. Everyone knows the truth now so this stance just makes him look stupid. It's like the kid with a mouthful of sweets swearing blind he hasn't taken any.

Nobody really cares anyway. Just admit what we all know anyway and move on.

[Einar Saukas]

Multiple sources have confirmed the same words from Lee Fogarty posted at his own private Facebook group, that I cannot access. I wasn't planning to quote him, but since this is now in the open, there's no reason for privacy concerns anymore. He apparently forgot to post his claims in his own forum, so let me help him:

Lee Fogarty: "I think it needs clearing up exactly what "created by" means. And another reason the data has to go is some of the changes are dubious or un-needed. This is why WoS always wanted to credit people submitting changes, and list them on the whats new page.
From what I can make out, the claim is that we are using the ZXDB database. Totally untrue - the WoS db was created a long time before - using the original data files.
Any group of people creating a database from an existing dataset will invariably create similar tables and structure. Things such as the machine types used - create a list for machine types.. both parties will likely create the same table with the same data. There is a WoS admins group on FB that Einar was in, and posts still there where I am sending structure/data to him.
That seems to be changing now to we are using "their" data. Again - untrue. There are some left over bits from a very old import test that are being removed.
WoS currently has over 300,000 indexed pages. Not just software. The software is a very small part of the database, and with the bits we are removing, comes to a minuscule amount.
This is all something that could have been sorted with a PM."

So that's the main point. Is new WoS simply using the same data from old WoS that was imported into ZXDB? Or is it using such an early version of ZXDB (from July/August 2016) that only contained old WoS data, so there's no need to credit ZXDB (despite literally about 50,000 fixes I did when importing this data)?

Unfortunately the answer is no. To understand the difference, let's take a look at ZXDB chronology. The summary below has plenty of links to prove everything, although I suggest ignoring the links for now and just reading from start to finish:

• July 2016: I started ZXDB to preserve WoS on July 3rd 2016, the day Lee Fogarty decided to quit. Soon afterwards, Lee gave me Martijn's old WoS internal files, asking Gerard and Martijn to help me understand them (thus I'm very grateful).

• August 2016: With their support, I finished converting Martijn's old WoS internal files to ZXDB, while also fixing thousands of inconsistencies. Last significant contribution from the WoS team to ZXDB content (Gerard sending me Martijn's answer to my last question about WoS files) was August 24th, 2016.

• In the following months, I continued to work on ZXDB adding new content and fixing lots of errors, with help from the entire community. Dozens of ZXDB versions were released, although they were not numbered.

• September 2016: Lee Fogarty decided he won't quit anymore. About ZXDB, he posted "I'll be using the cleaned data and converting to the WoS format" and "On the bright side, I don't have to test the import scripts as Einar has done that work".

• December 2016: Lee Fogarty changed his mind about ZXDB and posted "No - it's not compatible and we are too far down the line to change things". Then I posted "If you really cannot use ZXDB model, then please consider at least importing data from ZXDB instead of original WoS archive. Again, there are literally over a thousand inconsistency fixes and new additions to WoS incorporated into ZXDB. Why take a step back? If you fear it would be more work to import from ZXDB instead of original WoS archive, then once again I'm offering to do it for you.". But Lee Fogarty refused it too, providing several excuses (that I refuted). At this time, ZXDB already had a lot more content than old WoS.

• February 2017: After several attemps to convince Lee Fogarty publicly, I contacted him to discuss it privately. He asked me to join his WoS admin group on Facebook, so I created a temporary account (that I stopped using afterwards).

• May 2017: Three months later, it became too obvious Lee Fogarty had no intention to do anything. So I gave up and brought this discussion to the public forum again. Lee Fogarty has a different "recollection" of what happened, so I will say it again: I won't take the initiative of quoting private conversations with him without his consent, but if Lee Fogarty truly believes that's not what happened, he has my consent to publish these discussions himself!

• July 2017: Since Lee Fogarty continued to refuse using anything from ZXDB, I asked him to at least use the new ZXDB IDs from ZXDB, but he refused it too. It's kinda odd to mention it now

• November 2017: ZXDB was censored at WoS forum, so it moved to SpectrumComputing forum.

• March 2018: I was personally censored at WoS forum. I can still post, but I'm blocked from notifications when there's a response. I asked later for an explanation but never got one.

• April 2018: The official ZXDB site was created at Github and we start numbering new releases. ZXDB 1.0.0 was released on April 15th, 2018. At this time, ZXDB had a lot more content than old WoS, including a huge amount of corrections on the original content.

• September 2018: ZXDB version 1.0.8 was released on September 2, 2018. This is the ZXDB version that new WoS is using.

So that's the point. It comes back to something I wrote in my original post:

I imported old WoS content with their help from July 2016 to August 2016. If they had used one of the early versions of ZXDB that they participated, without crediting ZXDB, I would leave it alone. However they chose to use a version of ZXDB from September 2018, in order to take advantage of over 2 years of other people's work, without crediting anybody. That's a ZXDB version released 2 years after WoS stopped supporting ZXDB and started attacking my work. About 1 year after ZXDB and SpectrumComputing were censored at WoS thus forcing ZXDB to move to another forum. Months after I was personally censored at WoS without ever receiving any explanation.

Everybody is probably asking now, how do we know that new WoS is using ZXDB 1.0.8? Is it really much different from the original data from old WoS?

I'm glad you asked

I will have to get technical now, but I will explain it so everyone can understand. We will compare new WoS content against old WoS and a few ZXDB versions. Let's see what happens!

To reproduce this experiment at home, you need MySQL (or even better MariaDB) and any SQL client (HeidiSQL, MySQL Workbench, DBeaver, etc). They are open source and free. Also download a few versions of ZXDB from Github (click on "commits" to find and download older versions) and load one of them into your database.

I already mentioned you could visit new WoS software page and click on "EXPORT CSV (ALL)", to download some of the data from all titles stored in new WoS. I know lots of people did it (you may still have an old copy of this file yourself, perhaps in your Recycle Bin?). Let's start with "software-20200616.csv" from a day after new WoS was launched (we will talk about files from different days later).

Here's a database script to import this file into a database. Even if you don't know SQL, you should be able to see it's quite straightforward:

Code: Select all

create table x_newwos (
  rows0 varchar(100),
  id int(11) not null primary key,
  title varchar(500),
  slug varchar(500),
  no_players varchar(100),
  turn_type varchar(100),
  entry_type varchar(100),
  availability varchar(100),
  comments varchar(5000),
  is_x_rated varchar(100),
  is_crap varchar(100),
  clone_of varchar(100),
  old_id int(11),
  title_publisher varchar(500),
  publishers varchar(500),
  all_publishers varchar(500),
  entry_groups varchar(500),
  distribution_status_type varchar(500),
  display_image varchar(500)
);

load data local infile 'software-20200616.csv'
  into table x_newwos character set utf8
  fields terminated by ',' optionally enclosed by '"'
  lines terminated by '\n' ignore 1 lines;

Now download the original Martijn's WoS internal file "maindb.dat". Hopefully Lee Fogarty declared it "open source" so I don't need to worry anymore about sharing it. If you don't believe this file is authentic, choose any game at random and compare the corresponding line in this file against the old WoS pages. Let us know if you spot any difference!

Here's a simple database script to import this file:

Code: Select all

create table x_entries (
  titlekey varchar(500),
  pubkey varchar(500),
  title varchar(500),
  release_year varchar(10),
  orig_publisher varchar(500),
  re_publishers varchar(500),
  memory varchar(500),
  players varchar(500),
  joysticks varchar(500),
  genre varchar(500),
  category varchar(500),
  language varchar(500),
  distrib_status varchar(500),
  schemetype varchar(500),
  downloads varchar(500),
  flags varchar(500),
  authors varchar(500),
  aliases varchar(500),
  id int(11) primary key not null,
  spot_num varchar(500),
  spot_genre varchar(500),
  spot_full_price varchar(500),
  spot_budget_price varchar(500),
  spot_disk_price varchar(500),
  spot_comments varchar(500),
  spot_publisher varchar(500),
  license varchar(500),
  groupname varchar(500),
  comments varchar(5000),
  series varchar(500),
  orig_price varchar(500),
  c64_ref varchar(500),
  spanish_price varchar(500),
  wikipedia varchar(500),
  typein_ref varchar(500),
  authoring varchar(500)
);

load data local infile 'maindb.dat'
  into table x_entries character set utf8
  fields terminated by '\t'
  lines terminated by '\n';

The first CSV file from new WoS didn't have much useful content besides title, original publisher, and comments. Comparing title and original publisher from old titles won't help, since old WoS rarely got this information wrong so it almost never changed. However comparing comments is very useful, since they are continuously improved in ZXDB with fixes, further details, etc.

Here's a simple SQL to compare comments (except backslashes) between 2 tables. Notice it only compares titles that existed in old WoS (i.e 24369 titles with ID below 28187) to give new WoS a better chance:

Code: Select all

select e.id,e.comments,x.comments from entries e
inner join x_newwos x on e.id = x.old_id
where replace(coalesce(e.comments,''),'\\ ',' ') <> replace(coalesce(x.comments,''),'\\ ',' ')
and e.id <= 28187;

From this comparison, you will get the following results:

Code: Select all

new WoS (software-20200616.csv) vs. old WoS (maindb.dat)        - 2583 differences
new WoS (software-20200616.csv) vs. ZXDB 1.0.0 (April 2018)     - 5 differences
new WoS (software-20200616.csv) vs. ZXDB 1.0.8 (September 2018) - 0 (zero) differences
new WoS (software-20200616.csv) vs. ZXDB 1.0.9 (October 2018)   - 1 difference
new WoS (software-20200616.csv) vs. ZXDB 1.0.69 (latest)        - 766 differences

As you can see, there's a lot more similarity between new WoS and current ZXDB, than between new WoS and old WoS.

What if you want to repeat this test yourself to believe it, but you only have a newer CSV file from a different day? No problem. Although the CSV format at new WoS has changed over time, any CSV file downloaded before 2 days ago (when all comments changed into a bloody mess) will do. You just need to add or remove a couple columns from the import script, based on the column names you can see at the top of your CSV file. For instance, here's the same script adapted according to the CSV columns from 2 days ago:

Code: Select all

create table x_newwos (
  rows0 varchar(100),
  id int(11) not null primary key,
  title varchar(500),
  slug varchar(500),
  no_players varchar(100),
  turn_type varchar(100),
  entry_type varchar(100),
  availability varchar(100),
  comments varchar(5000),
  is_x_rated varchar(100),
  is_crap varchar(100),
  clone_of varchar(100),
  old_id int(11),
  title_publisher varchar(500),
  release_year varchar(10),
  search_title varchar(500),
  known_errors text(30000),
  has_inlay varchar(4),
  has_loading_screen varchar(4),
  machine_type varchar(500),
  publishers varchar(500),
  control_types varchar(500),
  theme varchar(500),
  all_publishers varchar(500),
  machine_types varchar(500),
  entry_groups varchar(500),
  az varchar(500),
  distribution_status_type varchar(500),
  display_image varchar(500),
  index x_id(old_id)
);

load data local infile 'software-20200702.csv'
  into table x_newwos character set utf8 
  fields terminated by ',' optionally enclosed by '"'
  lines terminated by '\n' ignore 1 lines;

The same comparison using a more recent CSV file will show nearly identical results, except for 2 titles: Reckless Rufus (new comments added on June 26th) and Werner's Quest (new comments added on June 17th but later lost).

It's absolutely clear that new WoS is really using content taken from ZXDB, not from old WoS. Instead of just pointing a few examples, we have now executed a comparison involving all titles. Even better, I provided instructions so anyone can replicate this experiment at home to see by themselves. And this comparison demonstrated that new WoS content is very much different from old WoS, not so much different from current ZXDB, and absolutely identical to ZXDB from September 2018.

As promised, this is my final post providing evidences. There's no need to prove anything else.

So what now? Well, THAT QUESTION will require one more post. But it's late, so let's talk about that tomorrow.


NOTE: On June 18th, I did a similar test at SpectrumComputing and found 2572 differences (instead of 2583) between new WoS and old WoS. It's because I compared new WoS against old Wos data that was already converted to ZXDB in 2016. This new comparison now, directly between new WoS and old WoS, is even more accurate and indicates even more differences.


NOTE: Reproduced from my post at the WoS forum

[Einar Saukas]

UPDATE: Please download "maindb.dat" from here.

[Einar Saukas]

SO WHAT NOW?

I have a few requests to the WoS team. Feel free to point out if I'm asking anything unreasonable at all:

• Please stop slandering me. JUST STOP. I moved away from WoS forum and I have been avoiding this subject because I'm tired and sick of this. But every few months, Lee Fogarty or Richard Chandler post false accusations against me. Every time, I come back here to reply with counter-proofs. And then silence. No apologies, nothing... A few months later, it happens again. If you look at my post history, you will notice everything I posted at WoS forum in the last 2 1/2 years (since ZXDB was censored and I left) fits this pattern (actually many posts before it happened fit this pattern too). Hell, I wouldn't have bothered to spend so much time writing this thread, to prove new WoS "stole" ZXDB, if only 4 days before new WoS launched, I had not been accused one more time of doing exactly the opposite. For the record, slandering me behind my back (private Facebook groups or other social media) doesn't make it any better either. Let's make it clear once and for all: Yes, Martijn's files were given to me. No, I didn't run away with it. Yes, Lee Fogarty made this data open source. No, I didn't receive anything from WoS admin group in 2017 then activated my own trustworthy portable time travel device to release everything in August 2016. Yes, everyone involved was properly credited in ZXDB. The reason I just wasted so much time in this thread is because I wanted to establish the truth crystal clear, so I won't get this crap again. THAT'S ENOUGH.

• An apology for slandering me would be the honorable thing to do now. Frankly I'm not really expecting it to happen anymore, but I'm an optimistic person!

• Please stop trying to disguise using ZXDB every time you are caught. It's not convincing anyone, it's just ruining everything. The line "Your honour, I'm refunding to banks the part of the money that the police has found, can I go now?" doesn't quite work. Now that it's proved new WoS is using ZXDB, anything else you change will be just "new WoS is using ZXDB but crippled". You won't get off the hook for using ZXDB dishonestly, and you will be providing a bad service for your own users. It's the worst of both worlds. A lot of users still visit WoS nowadays, either because they don't know any better, or because they just don't care. Everyone else that cares about your attitudes have already left, so you are just wasting your time trying to prove yourself to those left who obviously don't care. So please stop the whac-a-mole and focus your efforts to provide a decent service to your remaining users instead. Martijn spent decades building WoS credibility, his legacy deserves better.

• Please stop trying to take credit for other people's work. Nobody is questioning who built the new WoS site and has been maintaining the WoS forum. Is it not enough to receive well deserved credit for those? However the database is not yours. The old WoS files were built by Martijn, maintained by him for decades with help from many contributors, then converted by myself into a proper database (with my questions answered mostly by Gerard and Martijn), then maintained by the ZXDB team for the last 4 years with help from many contributors. As I mentioned already, new WoS only uses the list of publishers you converted in 2017 (everything else is taken from ZXDB) and it still contains the same errors that I pointed out in 2016 and you claimed to have fixed in 2017. Removing even more improvements from ZXDB 1.0.8 won't solve anything and won't benefit anybody. Please do the right thing, credit ZXDB properly at new WoS, as a sign of respect for all the people that have been working hard to keep Martijn's initiative alive. Notice there's absolutely no need to credit me personally for anything, I never asked for it, just credit ZXDB to properly recognize everyone's effort, and let's get over it.

• Please update new WoS to use latest version of ZXDB instead. Since you are already using it anyway, there's no point in keeping it outdated. It's bad for your own users to see wrong data already fixed everywhere else, and it's more work for the ZXDB team every time someone points out conflicting information between ZXDB and WoS, and we have to convince them that WoS is wrong.

• Please respect the ZXDB license. It doesn't exist to take "ownership" over old WoS data or anything. It simply asks to give proper credit to other people's work, and to retribute your usage of everyone's contribution by simply contributing back. It's not too much to ask, is it?

• Please stop trying to divide this community. In particular, ZXDB is a community effort, it will always remain open and freely available for everyone to use it honestly, including new WoS. An attempt from new WoS to compete, by taking an older version of ZXDB and then trying to update it independently, will prove nothing and it's just plain stupid. Instead, it would benefit everyone to work together and join efforts to improve it further. For instance, years ago Gerard was organizing magazines for WoS, so if he has anything to contribute with ZXDB now, it will be more than welcome. Likewise, the What's New page at new WoS shows that new WoS online update tool is getting used almost exclusively to add RZX files one-by-one, but a simple upgrade to latest ZXDB would already add everything automatically at once. However, if you don't want to cooperate, fine. As Lee Fogarty stated several times, data from WoS is "open source and fully available", thus from now on, I won't even bother to check if someone sees an useful update at new WoS and decides to send it to ZXDB too. Anyway, my offer for cooperation still stands, you know how to contact me publicly or privately, I'm still available, willing to help and make things work out. Either cooperate or leave it alone, it's your call. Just please cut the crap.

Now I have no intention to post about this problem again. Please don't drag me back.


NOTE: Reproduced from my post at the WoS forum

[Ralf]

That's fair what you are asking for Einar.

I would have yet another request, maybe this time not for admins but to some regular WOS fellows.

If you really don't care about all this buzz with ZXDB, then just don't take part in discussion.

I can understand that you don't care about databases, data accuracy, new releases, people's work,
one guy being unfair to another guy and you just want to babble a bit about Jet Set Willy.

I can understand that you lived last few years in a ground hole with a family of badgers and have no
idea about recent events.

But I can't understand you make emotional posts that you don't care. You know that you're contradicting yourself?
Leave it to the people who care.

[XTM]

This post by Einar feels like a cliffhanger at the end of a season of a TV series (like the two-parters at the end of a Star Trek series)

The ZX Files, S7E12 (Part 1 of 2), final lines of dialogue:

"Okay, Einar has posted. What's going to happen now?"

"Now we wait ..." (dramatic music as the camera closes in on whoever said that, then screen fades to black)

[Mike Davies]

Lee Fogarty from WoS Facebook Group:
...
From what I can make out, the claim is that we are using the ZXDB database. Totally untrue - the WoS db was created a long time before - using the original data files.

The claim is:

• the current version of WoS is showing data that wasn't in Martijn's WoS data files, but came from the October 2018 version of ZXDB.

We know this because comparing Martijn's WoS files and ZXDB of Oct 2018 shows the same differences. These are changes that are unlikely to have been made independently. For example, inserting an image into an entry's comments with a filepath that includes *zxdb*. How would WoS independently arrive at having a zxdb directory for images if it wasn't using ZXDB or referencing it?

Why would WoS refer to ZXDB ids of software (e.g. ZX81 titles that haven't been in WoS, and are currently not displayable in WoS) within the Spectrum ports of the title? Why would the WoS comment text on a title show ZXDB's raw linking format, instead of Martijn's WoS data file linked-reference style? And link to software with titles that don't exist in WoS, but do exist in ZXDB -- *and* use the ZXDB ids to that software?

Any group of people creating a database from an existing dataset will invariably create similar tables and structure. Things such as the machine types used - create a list for machine types.

Except, invariably the two independent groups wouldn't make the same mistake. But WoS made the same mistake as ZXDB: we know that ZXDB switched to using the machine type label *"ZX-Spectrum" (it was a mistake, based on an assumption that makes sense in Brazil), but why did WoS "independently" change from machine type label "ZX Spectrum" to the label "ZX-Spectrum"?

There are some left over bits from a very old import test that are being removed.
...
The software is a very small part of the database, and with the bits we are removing, comes to a minuscule amount.

There you have it folks, confirmation:

* Fogarty imported ZXDB into WoS "very old import test".
* Fogarty confirms that ZXDB-acquired data is in the current WoS because he is removing it as it's reported, and it "comes to a minuscule amount".

" No sir, I didn't steal that guy's money, I only have a miniscule amount of it"

So an import script for ZXDB into WoS did exist at a point in time, otherwise, how could a "very old import test" be done? If that import test was done in 2016/2017, Fogarty would have needed to take his time machine, pop forward to a date after October 2018, grab a copy of ZXDB version for October 2018, go back in time to 2016, so he could run ZXDB against his import script. I feel that's rather unlikely -- there's probably more interesting things you'd do with a time machine.

What's more likely to have happened is that Fogarty couldn't get Martijn's data files into his own SQL database. Either he didn't have the technical ability to get the import working, or he lost part of the data, and didn't have a working backup. So he borrowed an October 2018 version of ZXDB, and hoped no-one would notice. Because, he can't admit he failed where Einar succeeded, and he talked up the importance of his import scripts. And it's much easier to import data in an SQL database when the data dump is already a series of SQL statements.


Unsurprisingly, with the barest of scrutiny, Fogarty's claims don't hold water. That's why he doesn't respond directly to Einar's points, and prefers to reply from a safe distance away from the actual claims. Funny he doesn't respond to a claim on his own forum, in a section dedicated to discussion WoS.

[Rorthron]

There's another reason he took the data from ZXDB. The WoS files were hopelessly out of date, and it's easier to fix that by ripping off other people's work than doing it yourself.

To be honest, though, I'm not sure Lee's comments are worth this level of analysis. He's shown himself to be a transparent liar time and time again. No-one with knowledge of the facts and an ounce of sense would ever believe a word he says.

[Pegaz]

All this is true, but at the end of the day he is the winner.
He took ZXDB, ignores any attempt to resolve this matter and I’m sure he laughs at all of us and feels untouchable.
He simply doesn't care about anything or anyone, except for himself and his selfish interests.
No surprises, predictable and expected epilogue.
I hope Martjin doesn’t know about all this and has found peace of mind.
Martjin whom I remember, would never have allowed such a degradation of his wos.

[ketmar]

All this is true, but at the end of the day he is the winner.

nope, this is only a short-term win. the reputational damage is now irreversible. and that damage is not something that immediately bites you, but it will bite you hard in the future. "modern WoS" already lost alot of invaluable enthusiasts (and many of them are keeping the scene alive for decades!). as the time passes, "modern WoS" will lose more and more people interested in Speccy, and will become a forum used primarily for small talk. and there are alot of such forums in internet.

so in the end of the day, Lee will simply close "WoS", as it will become "too costly to maintain, and users are not really interested in Spectrum anymore".

all we should do now is just wait. no need to hurry, the time is on our side.

p.s.: no, i don't really want WoS to die. but in its current state i'd better see it closed. and it will happen sooner or later.

[Mike Davies]

He took ZXDB, ignores any attempt to resolve this matter and I’m sure he laughs at all of us and feels untouchable.

He can laugh as much as he likes. The community around ZXDB gets better every day, with no backwards steps. ZXDB is up-to-date, freely downloadable, multiple sites based on it, each pushing the envelope in different directions. There's still data out there that can be brought into ZXDB, so there's still interesting things to do.

That's something WoS had while Martijn was active, and after that WoS has lost that through inept and antagonistic leadership.

WoS being a broken 2-year-out-of-date implementation of ZXDB -- and going backwards -- isn't making a case for being the go to Spectrum archive on the Web. Is that really what the WoS community gets after a 5 year wait??

Thing is, all the praise for WoS is toward what Martijn accomplished. Fogarty took a successful and respected website and drove it into the ground. I'm astonished when respected people are suggesting it should just be shut down, for the sake of the community.

[Vampyre]

Well, Einar gave quite detailed arguments for his claims.
We will see soon what the response will be and who was right...

What are the odds that Lee will just ignore Einar's arguments and respond with some feeble and false whataboutery?

https://worldofspectrum.org/forums/disc ... -db#latest

[Guesser]

Wow, 850 words to not actually say anything.

[PeterJ]

Just to make a point that (this has been mentioned previously) Einar, Dave and I had our PM rights on WoS removed some years ago.

I did read through the post with hope, but it doesn't seem to answer any questions. It reads like something Dominic Cummings would have written for Boris. Maybe this could be clarified in the Rose Garden?

Even though ZXDB is Einars baby I did tell [mention]polomint[/mention] earlier in this thread that Lee or Richard were welcome to contact me. Lee is a member here and his PMs are not blocked. They have not made contact.

[Rorthron]

What are the odds that Lee will just ignore Einar's arguments and respond with some feeble and false whataboutery?

https://worldofspectrum.org/forums/disc ... -db#latest

Even by Lee Fogarty's standards that's risible!

[Stefan]

What are the odds that Lee will just ignore Einar's arguments and respond with some feeble and false whataboutery?

https://worldofspectrum.org/forums/disc ... -db#latest

OMG like watching an infant try to string some words together into an incomprehensible sentence, but then not one, but multiple paragraphs of gibberish. After reading it a few times I still don’t know what he is trying to say.

[RWAC]

"Ok, time to clear this up."

Then proceeds not to clear anything up!

[R-Tape]

I can't see it. Since yesterday, Malwarebytes has been giving a Trojan warning for main site, forum, anything. Just me?

[PeterJ]

I can't see it. Since yesterday, Malwarebytes has been giving a Trojan warning for main site, forum, anything. Just me?

Ok, time to clear this up. After god knows how many pages of discussions, nobody from ZXDB has contacted me at all.

Firstly, there will never be an agreement as to what happened with the old data. I say it was take in its entirety without permission, Einar says otherwise. Other have previously said they had the old database anyway. What they actually had was a very small selection exported to a CSV file from the main database. What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself.

Having said that, like I said, there will never be an agreement. I will however say that the WoS admins are very good at telling me when I am wrong - and in this case all have agreed and stated over the last couple of years that my opinion here isn't wrong. Everyone in the admin group were working (as far as we were concerned) on Infoseek. I should also point out here that our "secret" facebook group is the official WoS fb and currently has over 5,000 members.

So, moving forward....

Having spent so much time with the data, it was clear that there were issues with the database. Issues that were so numerous, the decision was made to rebuild it, using the initial titles as a starting point - people will know this from the old CSV download - there's a lot more to it than just that one file, but it was a starting point. We've posted numerous updates over the last 2yrs, detailing the work we are doing, and where we are at. To give you some idea, every single title and release has been manually checked. This has led to tens of thousands of changes. Not just checking titles - this has involved opening and checking every inlay and scan on WoS, commenting on price differences, advertised versions, etc.. It's been a mammoth task.

As we neared the end of this, I was also putting together the new screens. It was intended to put this live at the end of July. In preparation for new scans (around 300gb), I purchased a higher spec server. Don't forget - we are an archive. It's a lot different to just linking to archive.org and hoping it stays up. This means we currently have 630gb files on the server. Some - such as the new inlays - aren't available yet.

The server move proved to be as bad as the last time, with the old cgi files not playing well. These scripts are almost 25yrs old. The only ones remaining that hadn't been moved to the new website were the infoseek related ones. So, one evening I switched the new pages on. I made no secret that it wasn't the database we will be using, as it is still being processed (fingers crossed this week). I have said many time here, and on the facebook group that this data isn't accurate and will not be used when our new database is ready to go live. However, the way the WoS database is constructed means that we can take change requests - not all can be actioned until we have the new set of data, but a lot can be. WoS has 91 tables in the database.

For an example of how much has gone into this, the old database has 28,699 titles & versions listed. This currently stands at 33,014 - and we haven't completed compilations or mag tapes yet. This is still phase 1, as we also have thousands of new scans to go through - they will be processed after we have finished compilations. This won't mean much to the average user - who cares that there are 5 versions of Licence to Kill and not the 3 currently listed? However, for the likes of collectors, this is important information.

The Infoseek engine currently has 308,807 items indexed, plus over 300,000 indexed, downloadable files. This isn't restricted to software - we have indexed every magazine page separately - something that the old WoS db didn't do. Because of the indexing, it has improved the infoseek results.

As for the data - I appreciate people are waiting for the advanced search to come online, but as it stands at the moment, with pages and sections missing, we still have far more information available to download than ever before - and based on feedback, that can only increase. The API is comprehensive and provides a lot more than the older one, and is also still being built upon. We have built this data up with accuracy at the front at all times. I am sure there will be some mistakes in there - it's to be expected.

So, what now?

The old database is considered legacy. As of today, the old database can be taken and used at will. However, I would like all mention of myself & WoS removing. It is provided "as is" without any guarantee of accuracy or quality.

The correct database will be loaded in the next few days.
Post edited by Lee Fogarty at 6:59PM