How to proceed with "little bugs"

[8BitAG]

Can we have a conversation about the way to proceed with these "Little bugs" threads at some point please?

There are so many corrections in the earlier Little Bugs threads that have not been actioned, dating back over two years now.

It's all very well having a go at WOS about their out of date database, but I'm now finding it impossible to submit corrections here because the earlier corrections I've submitted (from over two years ago) haven't been added.

These long sprawling threads just don't seem to be working.

[R-Tape]

Can we have a conversation about the way to proceed with these "Little bugs" threads at some point please?

There are so many corrections in the earlier Little Bugs threads that have not been actioned, dating back over two years now.

It's all very well having a go at WOS about their out of date database, but I'm now finding it impossible to submit corrections here because the earlier corrections I've submitted (from over two years ago) haven't been added.

I think this is very unfair, and demoralising. I can understand the frustration at old fixes not being done, but your post only focuses on the bugfixes missed, almost implying that we've done nothing for 2 years. There have been a HUGE number of bugfixes done, but yes, they didn't start with the oldest.

We probably need more people doing the bugfixes. Are you up for it?

These long sprawling threads just don't seem to be working.

I don't agree with this either. These threads are a useful & easy to follow list, and the main problem is being able to keep up with it, not the format it is presented in. What are you suggesting instead?

[8BitAG]

I think this is very unfair, and demoralising. I can understand the frustration at old fixes not being done, but your post only focuses on the bugfixes missed, almost implying that we've done nothing for 2 years. There have been a HUGE number of bugfixes done, but yes, they didn't start with the oldest.

Well, it could be argued that it's demoralising to submit lots of corrections and have many of them not included. I'm not having a go, I'm just saying that the current system is not working for a variety of reasons. Yes, there have been many bugfixes done and other excellent work on ZXDB. Corrections from the "little bugs" threads do seem to have less priority than other updates; even when those corrections are addressing major mistakes in the database.

We probably need more people doing the bugfixes. Are you up for it?

Yes, you need more people doing bugfixes. I'm very much in agreement that people should put their money where their mouth is... My free time is already spent updating the archive and database over at CASA; along with various text adventure projects. I am happy to help out with little tweaks with regards to text adventures but I have quite a lot of existing commitments to the retro gaming communities.

These long sprawling threads just don't seem to be working.

I don't agree with this either. These threads are a useful & easy to follow list, and the main problem is being able to keep up with it, not the format it is presented in. What are you suggesting instead?

No, I'm afraid I can't agree that the threads are working in their current form. Not when there are minor corrections from the very first little bugs thread that still haven't been added. Not when people are starting to double post their corrections. That alone indicates that people feel their are issues.

What am I suggesting instead? Well, whatever makes the people who do the updates life easier. Would a corrections submission form on the database page work easier? Would a common format in each post be more helpful; making sure they contain all the various IDs and references to help the updater?

Would a separate sub-forum, with the update for each game getting a separate post, be more appropriate? Any follow-on discussions would then be captured alongside the original corrections... At the moment, follow-on discussions aren't shown with the original post... they could be a few posts,,, or even a few pages away. Even when a correction hasn't be made, users would be able to see easily that the mistake has been noticed because it would be mentioned in the corresponding game's thread.

I'm not having a go at the site or anyone who works on it. Just expressing my opinion that things aren't working most efficiently at the moment.

Do we need to trawl through the earlier threads, check what has been missed and amalgamate it?

Edit: I've just browsed the original Little Bugs thread... the majority of corrections are actioned, but every so often there is one that isn't... but there is no indication on that thread whether it has been added or not. I *do* think it would be worth revisiting and collating together the missing bugfixes from that first thread.

[StooB]

Can we have a conversation about the way to proceed with these "Little bugs" threads at some point please?

There are so many corrections in the earlier Little Bugs threads that have not been actioned, dating back over two years now.

It's all very well having a go at WOS about their out of date database, but I'm now finding it impossible to submit corrections here because the earlier corrections I've submitted (from over two years ago) haven't been added.

I think this is very unfair, and demoralising. I can understand the frustration at old fixes not being done, but your post only focuses on the bugfixes missed, almost implying that we've done nothing for 2 years. There have been a HUGE number of bugfixes done, but yes, they didn't start with the oldest.

We probably need more people doing the bugfixes. Are you up for it?

We already are doing the bugfixes! It often takes more time to find, research and report the bugs than it would to do the actual correction in the database. There does seem to be a bottleneck somewhere. How does the process currently work? Is someone manually updating the database entry by entry or are there csv files being imported and exported? Would it help if we gave actual sql statements to update the db?

[catmeows]

I must wonder - do you have any ticketing system to cover db changes or do you just scan various thready ?
I'm not criticizing, it Is just about my curiosity.

[PeterJ]

Hi [mention]8BitAG[/mention],

Ones which are fixed or in process of being fixed should always have the Reviewed Icon.

✓ Reviewed

If you find ones that are fixed but don't have the Reviewed mark, PM me with a list and I will get them added. I would agree with Dave that as long as the Reviewed system is used, then this is the best way to do it.

I would also say they should be done in order (there will always be exceptions obviously when something is urgent, or needs further investigations).

Can you do SQL [mention]8BitAG[/mention]?

Its balancing the limited time between website updates, adding new features, adding new titles, adding screens and bug fixes.

[PeterJ]

We already are doing the bugfixes! It often takes more time to find, research and report the bugs than it would to do the actual correction in the database. There does seem to be a bottleneck somewhere. How does the process currently work? Is someone manually updating the database entry by entry or are there csv files being imported and exported? Would it help if we gave actual sql statements to update the db?

Hi, Dave means to check the bug fix is appropriate and create the SQL to update the database. If you want to be involved with this please PM [mention]Einar Saukas[/mention].

[8BitAG]

Ones which are fixed or in process of being fixed should always have the Reviewed Icon.

✓ Reviewed

If you find ones that are fixed but don't have the Reviewed mark, PM me with a list and I will get them added. I would agree with Dave that as long as the Reviewed system is used, then this is the best way to do it.

This is not the case for that first "Little Bugs" thread.

[PeterJ]

Ones which are fixed or in process of being fixed should always have the Reviewed Icon.

✓ Reviewed

If you find ones that are fixed but don't have the Reviewed mark, PM me with a list and I will get them added. I would agree with Dave that as long as the Reviewed system is used, then this is the best way to do it.

This is not the case for that first "Little Bugs" thread.

Hi, that is why I said 'If you find ones that are fixed but don't have the Reviewed mark, PM me with a list and I will get them added'

[PeterJ]

Will you volunteer [mention]8BitAG[/mention] to collate remaining issues from the first little bug thread? Just wait until the next database update though, as there has been a lot of work in recent weeks.

[PeterJ]

[mention]8BitAG[/mention]

Ive spent 10 minutes just sample checking some of the posts in the first little bugs thread, and most of the ones I have looked at are done so I have added the Reviewed check box.

Can you confirm if you are able to help with this [mention]8BitAG[/mention]?

[Rorthron]

I think there were some problems tracking fixes in the first little bugs thread. I know some of the fixes I pointed out were never made. And I understand that it can discourage people from filing corrections if they feel they are not being incorporated into the database.

However, since the "reviewed" box has been introduced, the system seems to me to have worked quite well. I haven't noticed any corrections being missed. There is a backlog, but this isn't anyone's full-time job, so it's inevitable. Unless anyone else gives up their time to make the updates, we just have to be a little patient. I am also aware that progress has been slowed recently by the fact that I dumped a huge amount of credit info in the cover artists thread, but [mention]druellan[/mention] seems to have got through that now.

So for new corrections, I don't think we need to change anything, except have more volunteers (not me, I'm afraid). But for the missed corrections in the old threads, perhaps we could have a thread of "missed little bugs", where we can flag any that have been missed before the "reviewed" box was introduced. (To be clear, though, I don't think we should be duplicating unactioned items in the new threads, where they are just part of the backlog and where there is a system for tracking them.)

[PeterJ]

[mention]Rorthron[/mention]

[mention]druellan[/mention] has indeed done a splendid job.

I don't think we need another thread. Lets just focus on the first little bugs thread initially and PM me on one list the ones which have been actioned and I will add the 'Reviewed' Sticker.

[8BitAG]

Ive spent 10 minutes just sample checking some of the posts in the first little bugs thread, and most of the ones I have looked at are done so I have added the Reviewed check box.

Can you confirm if you are able to help with this @8BitAG?

I could look at this at some point in the future once I have cleared some current archive research projects. It does seem a bit easier for you (or another moderator) to do as you can actually access the posts directly and add a reviewed box.

If I did it, I would have to laboriously copy and paste the information and links from any unactioned threads into a new document or post. There would be very little point in my flagging up the posts that have been actioned.

[PeterJ]

Hi,

Just the URL for the post for the ones which are done, but not flagged is fine. Nothing more. You just right click on the post title and choose copy link, then just post it into Notepad or similar. I did 5 in about 12 minutes. I will then go in and flag them for you if you spend a couple of hours going through them.

Its entirely up to you, but it would help get that first thread cleaner as you quite rightly desire. We will then have a clear picture of the ones not done.

There would be very little point in my flagging up the posts that have been actioned.

This is exactly what is needed. We then know the ones to be focused on.

[Rorthron]

@druellan has indeed done a splendid job.

Agreed.

[Einar Saukas]

First of all, I think this is a very valid concern. It also worries me that there are still so many pending bugfixes to be processed!

I would like to point out a few things:

• There's a huge backlog of accumulated problems. The old WoS archive stopped regular updates in 2012 (with just a few new titles added in 2013). For over half a decade, many old problems were identified, also a lot of new content was produced or discovered. Everything that's either wrong or missing from all these years is currently reported as bug.

• Although I implemented myself many thousands of fixes since ZXDB started, my efforts were concentrated on inconsistencies within the archive itself. Also I was dividing my time between too many tasks (evolving ZXDB structure, integrating with other databases and sites, developing ZX-Ed, etc). A concentrated effort to process all these accumulated problems only started a year ago, when [mention]druellan[/mention] took over this task. He doing an amazing job, but let's not forget he's also dividing his time with other things. For instance, we currently have 2,875 author roles in ZXDB, [mention]druellan[/mention] added most of them himself and these are not bugfixes since old WoS never had this information!

• Every ZXDB update contains dozens of fixes. We are currently fixing bugs at a faster rate than new bugs are reported. If you take a quick look at our bugfix threads, you will notice that most reported bugs are fixed already. Even so, there's still a lot of work to be done. The huge backlog I mentioned means it's still going to take several months at current speed before we catch up.

• We only started marking posts with a "Reviewed" icon quite recently. This idea was actually refused the first time I suggested it. For this reason, many old bugfixes were already done but not marked.

• Old bug reports are not neglected. For instance, only 2 days ago I posted a request for further details in a bugfix thread that started over 2 years ago. We are simply trying to process them all in whatever order will make our work faster. There are also other concerns, for instance sometimes there's a bug report that I know will be giving more work for Dmitri to import next ZXDB into ZX-Art if I don't fix it quickly, so I try to do it first. Sometimes I skip bug reports about incorrect magazine references, because it will be more efficient to process them all together later. And so on... My apologies if that looks confusing, but our final goal is to have all bug reports processed ASAP, without concern for the order they were reported. So don't worry if an old bug report is not done yet, it won't be forgotten. That's exactly the reason we are using the "Reviewed" icon, to ensure nothing will be missed.

• We don't need anybody to provide SQL instead of bug reports. Nobody is writing SQL manually, both [mention]druellan[/mention] and I use a tool called ZX-Ed to edit individual bugfixes (that automatically generates SQL for us). And when we need to add a lot of new information at once (for instance indexing all issues of a certain magazine), I usually ask people to fill a spreadsheet and I create a script to generate SQL automatically from there, because it's faster to do it this way for bulk data. Therefore SQL is not a problem at all. The reason bugfixes take time is that [mention]druellan[/mention] and I need to double-check every bug report (obviously nobody is infallible so we cannot just blindly apply every change that was reported), sometimes we need to post asking for further clarifications, sometimes we need to discuss with each other the best way to add or change certain information (many changes are not as straightforward as it seems!). Quite often, making the actual change in ZXDB is the easier part.

• I don't think we need a new system to report bugs. In my opinion, this current system is working well. I have a few ideas to improve it (I will write another post about it), but overall the main issue is the huge amount of accumulated backlog over the years, not the way we are handling it. As long as we keep going the same way, it shouldn't take too long to get everything sorted

[PeterJ]

For this reason, many old bugfixes were already done but not marked.

As mentioned this needs doing retrospectively otherwise we won't know what's done and what's not.

This is where I was asking for a little assistance.

Agreed with [mention]Einar Saukas[/mention], we don't need yet another system.

[Stefan]

What was the reason for putting /all/ little bugs in one thread? Wouldn't a thread per bug be a lot simpler? Especially when some of the little bugs spiral off into further discussion?

[PeterJ]

Hi [mention]Stefan[/mention],

They are spread over three threads (soon to be four). I don't think a thread per bug would be any benefit.

We would prefer that threads didn't spiral off, but you are right sometimes they do!

[8BitAG]

There would be very little point in my flagging up the posts that have been actioned.

This is exactly what is needed. We then know the ones to be focused on.

I'm sorry, but I would find it hard to be motivated enough to tag things that have already been fixed, knowing full well that somebody else would then need to trawl through the same threads and posts an additional time to spot the things that haven't been updated.

It would be much better to lock off that thread, thoroughly identify the fixes that still need to be made and document them. Then those very old fixes can be addressed as a priority and Little Bugs threads 2 and 3 can proceed using the current system.

[PeterJ]

No problem [mention]8BitAG[/mention]. I've started working through it already, and marked 10 as processed. We will then know which still need fixing or at least reviewing.

[Einar Saukas]

OK, I will post now some suggestions for improvements. Please let me know your thoughts:

• It really REALLY helps when bug reports contain proper links. If you mention a game without providing a link, we have to search for this game, and we may even pick the wrong one when there are multiple games with the same title. If you mention a magazine article, providing a link to this article (or a link to this magazine issue webpage in SpectrumComputing) helps a lot. If you mention an author or company, please also provide the corresponding link in SpectrumComputing. Most bug reports already provide these links (thank you very much!) but I just wanted to enforce how much it helps!

• Forum section "ZXDB Discussion" currently mixes threads about bugfixes, and also threads about other ZXDB discussions (how to install or use ZXDB, etc). I think it would help to create a new forum section called "ZXDB Fixes" (within "Archive"), then move all bugfix threads there. Of course [mention]druellan[/mention] would be moderator of "ZXDB Fixes" (so he can continue tagging reviewed bug reports) instead of "ZXDB Discussion".

• Whenever all posts in an entire bugfixes thread has been processed, perhaps we could simply lock this thread. It would make it easier to keep track of threads that still contain bug reports to be processed, or not. Later, if someone ever finds out new information that needs to be posted, the person could either start a new thread, or contact one of the forum moderators ([mention]PeterJ[/mention], [mention]R-Tape[/mention], or [mention]druellan[/mention]) to unlock the thread. I know this is a radical idea, but I'm concerned about threads like Cover artists, that currently contains 420 posts and [mention]druellan[/mention] processed almost all of them, but very few posts are marked because this thread started when the "Reviewed" icon didn't exist yet. It should be easier to start a new thread "Cover artists 2", let [mention]druellan[/mention] finish processing the original thread, then lock it. Although I'm not so sure about this idea... What do you think?

[PeterJ]

[mention]Einar Saukas[/mention]. People don't tend to post in the old little bugs thread, once a new one is created. I don't see the benefit of locking until all the issues are fixed, but I'm not strongly against the idea.

I would agree that once every issue in a thread is dealt with it can then be locked.

I also agree about including a link to the title in a bug report. I raised this earlier in the week when it caused an issue.

Every fixed item needs the reviewed icon retrospectively applied.

All 'easy fix' issues should be fixed in strict rotation. This includes and starts with the ones in the first thread.

[Einar Saukas]

Here's a quick list of fully "reviewed" threads that could be locked now (if there are no objections):

viewtopic.php?f=32&t=461
viewtopic.php?f=32&t=600
viewtopic.php?f=32&t=619
viewtopic.php?f=32&t=618
viewtopic.php?f=32&t=649
viewtopic.php?f=32&t=782
viewtopic.php?f=32&t=916
viewtopic.php?f=32&t=958
viewtopic.php?f=32&t=1002
viewtopic.php?f=32&t=1063
viewtopic.php?f=32&t=1119
viewtopic.php?f=32&t=1313

There are probably others.