Some AIs are crashing all the time!

[Wuzzy]

One thing that frustrates me about the community AIs is that many of them are broken. They crash all the time, it's not fun.
There aren't many AIs that do not crash.

What happened to basic quality control? It's so bad, if you download a random AI and play for 15 minutes, you have a good chance you get a crash. It has gotten completely out of hand.

I think some drastic measures need to be applied: I suggest to remove ALL AIs that are known to be broken from the "in-game extension downloader", alert the developer and tell them their AI is only going back when the critical bugs have been fixed. The official "in-game extension downloader" should not be a free-for-all for every possible add-on, no matter how broken. That's just terrible for the players.

What would also help a lot in discovering ton of bugs if the AIs would be automatically tested to check for common bugs or rare corner cases. Also performance test on extreme cases like 4096×4096 with many towns and industries.

[Wuzzy]

Here are some quick test results in OpenTTD 1.9.3:

Crashed either instantly or after only a few minutes:
- WrightAI v3
- TracAI v1
- AIAI v97

(too lazy to post crash logs, just test your scripts!)

Places bad stations:
- AIAI v97 (random bus stops in the middle of nowhere, no attempt to connect to road)

Failed the 1 town test:
- CluelessPlus v38
- HeliFerry v4
- TeshiNet v4
- BorkAI v23
- AIAI v97
- OtviAI v422
- SimpleAI v14
- Trans v190723
- AdmiralAI v25
- trAIns v2
- SynTrans v15
- PathZilla v6
- Terron v209
- DictatorAI v183
- ChooChoo v412 (places sign "ChooChoo: BuildNewNetwork" at random locations)
- Chopper v10 (places 1 unused airport)
- CivilAI v23 (builds 1 street depot and unused bus stop)

The "1 town test" is a game with only 1 town and no industries. Not a single AI I have tested is able to figure this one out. The solution is to build two stations in the same town.

Doing nothing nothing:
- If ChopperAI starts in a year in which the helicopter was not invented yet, it does nothing. It would be nice if starting the company could be prevented when it's clear the AI couldn't handle it.

AIs that place annoying signs by default (!):
- OtviAI v422 (sign at HQ)
- CluelessPlus v38 (sign at top of map with random nonsense)
- Terron v209 ("Road expansion: 0%", "Rail expansion: 0%")
- AIAI v97 (4 annoying HQ signs, disabable)
- ChooChoo v412 (disabable)

(IMHO there should be a global setting to prevent AIs to place any signs, I don't know why AI coders like to spam signs.)

[Redirect Left]

There are definitely a lot of AIs out there that rarely function as intended. Unfortunately also a lot of AIs get abandoned, sometimes relatively fast, by authors who either move on to other stuff or stop playing Transport Tycoon.
I've tried to look into doing one myself with all my spare time and ability to update stuff all the time as bug reports come in, but for some reason the language AIs use is completely beyond my understanding. I know several programming languages but for some reason whatever AIs are using just makes no sense to me compared to things like C, LUA & Javascript.

Most AIs work 'well' only under very specific conditions, and if the map doesn't have X or Y or the year is Z instead of A, the entire thing has a heart attack and acts like Linux has kernel panicked or Windows has blue screened on it.
AIs also often have very stupid ways of dealing with some routes, as detailed here where the pathfinding looks like it was done by a blind drunk horse.

(IMHO there should be a global setting to prevent AIs to place any signs, I don't know why AI coders like to spam signs.)

They're often there so the player (or developer) can look at what the AI is thinking of in real time. Although most of them (should) clean up the signs after they've finished doing whatever they were signposting. For example, the one you mentioned 'ChooChoo: BuildNewNetwork' is showing you the AI is looking at building a new network, i would presume at that location, and then when it has finished deciding, it should delete it and either not build it or build it there.
When used properly, allowing AIs to build signs is very helpful overall.

[Timberwolf]

This is a complex problem with the content system. A lot of those AIs are unmaintained - the authors have left the community, or no longer have the time to maintain them, but the content stays on BaNaNaS for savegame compatibility and the general good practice of digital preservation.

Many of those used to work, but as OpenTTD has moved on they've not kept pace with changes to the game. You also have to consider these are volunteer projects. It's quite possible the authors don't play with corner cases like a single town or a very large map, and would rather spend their time elsewhere. For example, with my game script Villages Is Villages I do my best to handle the strange cases people have made bug reports for over the years, but it's still a pain downloading several 100MB of (often hard to find) newgrf files to track down some weird corner case. I mostly do it because people have asked nicely and sometimes the technical challenges of why it's broken are interesting even if I don't care for the gameplay setting.

I do think there is an issue with old, broken content, duplicates of the same file for minor version changes, and a fair amount of low effort stuff. Conversely there's also a big problem with, "I decided I didn't like the content system, so now you can only play on this server if you can hunt down the Reddit thread this specific version was posted on, good luck". If you tweak one you worsen the other. In my case, if there was some sort of approval process or formal review I wouldn't have uploaded Villages Is Villages or Timberwolf's Road Vehicles, which means you also wouldn't have Trains, Roads or Stations. (Some might argue this would have been a good thing).

How to fix it? I think maybe culling some of the things which hard crash for everyone on a version-by-version basis would be useful, behind a filter so you can still download them and play around if necessary. But that needs in-game telemetry and then some kind of abuse protection, unless someone manually has to check and approve each crash report. And it doesn't solve the "it's broken for me, but perfectly fine for someone else's use case"

This is also as good a place as any to mention I'm finding this strident "OpenTTD is totally broken and all the community content is awful and it needs to be fixed NOW" style of problem reporting rather grating. It's an excellent way of discouraging people from making anything. Certainly if the people who'd reported the first problems with Villages Is Villages on 4096x4096 maps had done it in a "this is terrible for all players and it should be removed from the content downloader until it's fixed" style I'd probably have stopped bothering uploading content for anyone else to use.

[Wuzzy]

I don't understand why AIs need to spam signs in the first place. If it's only for debugging, it should at least be turned off by default. And if it's about "remembering locations", couldn't the AI just save those internally? I see no reason why AIs need signs ever, except for debugging. The signs that AIs use are rarely meaningful to the player, anyway. I think AIs should generally not use signs and if they want to, only use them if turned on by the player (in the AI settings). I normally do not want to see what the AI "thinks" while playing the game, and I suspect most players think so as well.

AIs should not place player-visible signs by default, unless there's a really good reason. Anyway, that's just my opinion. Maybe there are some instances in which AIs add useful non-debug signs without being annoying. I just haven't seen that yet. Feel free to show me an AI that does.

AIs that are no longer maintained would not be an issue if they would at least not crash. P

Don't get me wrong: I'm all for messing around, and experimenting with AIs, and building lots of crazy things, and I do not want to take that away. But I'm talking specifically about the OFFICIAL add-on system, i.e. the thing that is visible to the average NON-TECHNICAL player. It makes OpenTTD look bad if the majority of AI scripts is broken. There needs to be at least a bare minimum of quality assurance, like "no crashes". You can experiment with scripts all you want, but "getting approved to the content system" should have at least some requirements. Like "no crashes". There might be some grace for scripts that crash in only very academic, theoretical situations (although this still needs to be discouraged, obviously), but there should be no mercy for scripts that reliably crash within 5 seconds or under common circumstances.
I also don't buy the compability argument. Compability really does not matter if the AI does not work in the first place.
You might say I come over as a little rude, but to be fair, the problem is serious if things are so broken. It's very bad for gameplay experience when an AI crashes, it ruins the game. You can restart the AI, but this will destroy the company, including everything the AI has built.

To clarify the "no crash" requirement: No crashes under default settings and without any NewGRF active. So the absolute minimum. I admit that testing an AI with every possible NewGRF combination is not a reasonable requirement, it would be too much, sadly.

I'm not saying that every AI is "completely broken", nor do I want to say that everyone is just a fool. But I do want to show a systemic problem from a purely player perspective.. The problem is not even that we have 'broken scripts' but that that the broken scripts manage to get their way to the player, without any noticable content curation. The apparent lack of curation is the problem I want to talk about.

I also think that OpenTTD needs a stable, high-quality, general-purpose, well-tested default AI. Currently, OpenTTD ships with no AI at all, and AI development is not part of “official” development either. Which is … sad. If OpenTTD had just one reasonable default AI, the current problems with the community AIs wouldn't be that big of a deal anymore, because then there's always an AI to fall back on.

To summarize, this is what I suggest:
- Refuse AIs from being shown in the content manager if they are provably highly unstable (crashing a lot under common circumstances even w/o NewGRFs)
- Add the AIs back when the worst bugs have been dealt with

When I think about it, the problem is more about the apparent lack of content curation, and less about the scripts themselves.

[2TallTyler]

+1 to everything Timberwolf said.

One potential improvement to the content download system might be a star-review system where users could rate content, allowing good stuff to float to the top and things which no longer work to sink out of sight. A good example is the Steam Workshop for a game like Cities: Skylines. It would also help with the constant questions of, "I'm new, what NewGRFs should I get?" (which is an understandable question, as there isn't really a good way of seeing what's available). But extending the UI and database to add such a feature is a big project.

[Wuzzy]

User rating systems are subject to brigading and abuse, it won't fix the problem of bugs. It's also highly subjective, because players have different expectations from AIs. Players might give low rating simply because they don't like the playstyle, yet the AI is still perfectly functional. Also a star rating system doesn't really tell you how stable a script really is. It will also be hard for script authors to "redeem" themselves when they have completely reworked a script simply by fixing all bugs, because by that time, most of the ratings will be outdated, and people rarely "re-rate" things. In short, I don't think a rating system will solve any of the stability problems, you're just outsourcing the software testing to the players (remember we're talking about the OFFICIAL in-game content system, not a random forum thread, players have a reasonable expectation that stuff Just Works™).

Scripts that are known to crash a lot should not be visible to normal players in the first place. Minimal content curation is the answer. Maybe a simple reporting system would be better. Players report broken scripts and if it's confirmed to be real and serious enough, it is pulled & author is alerted.

[Timberwolf]

Well yes, someone should build a stable reference AI. A laudable aim, which I have toyed with and as a result mostly understood why this hasn't been done. This is one of the many things which irritates me about these thread storms of "what's wrong with OpenTTD and needs to be fixed", they seem to start from the premise that nobody has ever noticed these issues in (say) the 12 or so years NoAI has existed, but thankfully now the hard work of discovering it has been done it only needs someone to do the easy bit of actually fixing it.

The problem with curation and testing is someone has to take the time to do it. An organic solution like star ratings would be the most elegant, as it doesn't require a thankless task to become someone's (that word again) job. Maybe some sort of "AI CI" that runs the AI on a dedicated server for a bit before you can upload, but... ah, someone has to pay for the computing resources and we're back there again. I'm not sure if it's particularly feasible with the current BaNaNaS infrastructure to do such a rating system, but I haven't looked too closely at it. You'd also need some way to handle "this used to work, but now it's broken in 1.10.x" situations and also the challenge that new sets come along all the time. I'm kind of against strong curation as funnelling people to a small number of existing sets rather than giving a fair chance to all authors, even though I see a benefit to new players in providing a few core combinations like UKRS2+HOVS+BATS, "full Andy" and so on that work well together. It'd also be nice to hide all of the beta versions, old abandoned sets, and things without going to the extreme step of the blacklist used for ripped-off sprites and spam.

Another challenge is a lot of stuff that's now broken used to be widely used, downloaded thousands of times and in theory likely to have gathered hundreds of 5* ratings. In the AI case, plenty only need simple fixes to handle some of the new game concepts that have been created, but sometimes it's difficult to contact the authors to get those uploaded. For example I've fixed a couple of bugs with CivilAI for my own use, but it's not my AI to re-upload and Pikka isn't around any more to implement those changes. Re-uploading carries the risk of licence violation for non-GPL or non-CC stuff, and of course worsens the problem of "now there are two versions of this AI, which should I use?" facing new players.

It's a bit like the serious base underlying my rather tongue-in-cheek comments about "the price for complaining about extra zoom sets is you have to make an extra zoom set" - just once it'd be nice to see this kind of thing expressed as "I made a PR to allow hiding AI-built signs in the transparency options" or "I've started an attempt to build a bug-free reference AI, please download and test", i.e. actually doing something about the problem rather than just complaining about it as if no-one's ever spotted it before.

[Wuzzy]

Hiding ancient alpha or beta releases or versions that have been long superceded would be a nice start indeed. It could be even up to the author to define what a "beta release" is. Although I don't think that proliferation of old or duplicate versions is really serious at the moment. It's a real problem, but not really one I think is that big of a deal. It could be worse.

I see now the problem that manual curation needs a curator … Yeah, that's a problem indeed. Especially if those people disappear.

I wonder how content manages to get into the content system … Does it need an approval from anyone? Or can everyone just post an add-on and it instantly appears for everyone? I mean, is there any verification required?

I am still not convinced the star rating system is going to cut it. You have basically mentioned all the problems with star ratings yourself. All reasons why I am skeptical. It's not a great solution.

Re-uploading carries the risk of licence violation for non-GPL or non-CC stuff, and of course worsens the problem of "now there are two versions of this AI, which should I use?" facing new players.

Well, since OpenTTD promotes non-free software I'd say that's kind of a self-inflicted wound then. The solution, of course, is “boycott all proprietary content”. Also note that not even all CC'd content is really free/libre, there is sometimes the awful "NoDerivs" or "NonCommercial" clause which is incompatible with free software licensing. Yeah, I think just let those add-ons rot. They don't deserve support from the FOSS community.

Talking about automation: Some kind of automatic testing would be really great. I mean, I do know that community AIs have many bugs, but I can't really put my finger on where exactly the bugs lie. Automating the testing away, even just partially, would probably help a ton finding all those "hidden" bugs. But I admit I have no idea how exactly that would work in practice. I'm not sure if even a dedicated server is really required to do with. On the other hand, where are all those donations going to, anyway?

Hmmm, I am out of ideas right now. To be honest, I can't really think of a great solution on how to solve this once and for all. Curation was my main point here, but if that's not possible, I guess we're doomed.

Maybe the only path forward here is give up on the problem altogether and write a big fat notice that every add-on you see might be completely broken and that the OpenTTD team takes no responsibility whatsoever.

What do you think about taking/forking SimpleAI and turning it to the "official default AI"? Heck, even if SimpleAI is included into official OpenTTD verbatim (except renaming it to something like "DefaultAI"), that would already be a huge win. I think SimpleAI is "good enough" for now. It's closest to TTD behavior, it builds the basic things OK enough, it's GPLv2'ed. The only downside for now is that it doesn't support ships. But that's still so much better than nothing.

I think the simple act of adding any default AI would be a massive improvement of OpenTTD that cannot be understated. Not really because SimpleAI is that great, but because it would be the first time in years since OpenTTD shipped with an AI (the last time was in 0.7, I guess, i.e. ages ago). So in ther words, the day OpenTTD adds a default AI would be historic. xD