OTTD Unlimited?

[richk67]

Wait, you mean I have to learn lua if I want to do anything that the xml spec author hasn't predicted I'll want to do? How come "every child" can write languages you're familiar with, but not nfo?

Code: Select all

1 * 1  02 00 01 81 0A 04 01 01 <sound> 00 00 <fail> // play sound every second event
1 * 1  02 00 01 81 10 00 FF 01 01 00 07 07 <fail> // sfx event check (16 tick)
1 * 1  02 00 01 85 0C 00 FF FF 01 01 00 33 00 33 00 <fail> // callback check
1 * 1  02 00 01 85 B4 00 FF FF 01 01 00 60 00 8E 00 <fail> // speed 60 to 89

Ermm... that isnt a LANGUAGE. That is hex code. A language would have human readable tokens that are translated to the target code by a compiler/interpreter. NFO is just hex - even lower level than assembler.

[Haukinger]

Who cares which is faster ? Who knows in advance if a carrier pigeon specified in an extended version of nfo or in lua is actually faster in a given situation without having tried out ?

Is speed of execution the only agrument ? Isn't that totally outweighed by being a hundred times easier to write, read and maintain ? After all, using a constrained environment is much safer than allowing thirdparty code to be executed directly on the cpu.

[Haukinger]

Ermm... that isnt a LANGUAGE. That is hex code. A language would have human readable tokens that are translated to the target code by a compiler/interpreter. NFO is just hex - even lower level than assembler.

It's a very special kind of a language... with very non-intuitive tokens, such as 0x02 or 0xff (that constanty shift meanings when appearing at different positions) and very un-understandable semantics
Like talking of java-bytecode as a language (which it surely is, but a damn weird one)...

[Brianetta]

Is speed of execution the only agrument ? Isn't that totally outweighed by being a hundred times easier to write, read and maintain ? After all, using a constrained environment is much safer than allowing thirdparty code to be executed directly on the cpu.

Performance is possibly the most important argument, but I'm not close enough to the development to say. Newgrf is a very busy component of TTD in both its forms.

[Haukinger]

From an architects perspective, if performance is your main problem in this 100 KLOC mess of spaghetti code, you have a *real* problem here

[Patchman]

I believe that the point is, currently there isn't a problem with performance because it's so tightly integrated.

If you go to a script-based architecture, performance will become the bottle neck, because you have to run the sprite calculations for every vehicle at every engine tick.

This is because sprites can change based on speed, based on direction of travel (the Osprey changes its rotor direction), and even (for steam engines with animated rods) just for every tick. The nice thing with NFO is that it doesn't have to care what crazy ideas the grf programmer wants to come up with, because it's fast enough to just let him do these things without having to "think" much.

Sure you can optimize some of these calls away, but even then in a large game there may be tens or even hundreds of thousands of newgrf evaluations per second. Do you really think that a scripted language can be fast enough to cope with that? The only alternative I see is to somehow hardcode a way for the grf file to tell the scripting engine when the sprites are allowed to change, and only call the sprite evaluation in these cases. This however is very limiting, because you need to add a hook for every possible use case, i.e. add a hook for re-evaluating sprites when the speed changes, when the direction changes, when the aircraft altitude changes, etc etc.

To make a scripting engine even remotely as fast as NFO would require it to be far more restrictive.

[Haukinger]

I believe that the point is, currently there isn't a problem with performance because it's so tightly integrated.

If you go to a script-based architecture, performance will become the bottle neck, because you have to run the sprite calculations for every vehicle at every engine tick.

Well, if that is so, you have to do it now, too. The good thing is, the cases in which things need to be changed are rare. If you have lots of steam engines with animated rods, you can slow a core2duo to a crawl eventually...
Am I right to assume that the animated rods aren't actually rendered each tick, but a frame from prerendered animation is displayed (the callback returns a sprite-number) ?
Anyway, how many games do you have that are actually that large ? I mean, what gain do you buy with the complicated nfo-language ? I understand that some years ago computers were slower, but they have advanced. And are still. It doesn't look to me as if ottd will be final within the month or even year, so why not plan for faster computers ? If a lua-newgrf limits you to 300 trains per game today, it will limit you to 1000 or more in two years.
Besides, there's a lot of room for optimization that doesn't require having a wannabe scripting-language that's understandable only by experts. See this thread for example... I doubt anyone can safely predict the runtime behaviour of a large system like ottd.

[Rubidium]

If a lua-newgrf limits you to 300 trains per game today, it will limit you to 1000 or more in two years.

That assumes that OpenTTD becomes multithreaded as the amount of instructions per clock per CPU executing unit[1] has not been increasing that much the last year. Because OpenTTD is single threaded and as Intel and AMD are only focusing on increasing the amount of instructions per clock per CPU by increasing the amount of executing units it will not cause OpenTTD to run more than 3 times as fast in two years.

[1] whatever it should be called; here I mean Core2Duo has two executing units, a HyperThreading processor has two executing units, a QuadCore processor has 4 executing units, a SMP with two single-core-non-HT processors has 1 executing core)

[Patchman]

Well, if that is so, you have to do it now, too. The good thing is, the cases in which things need to be changed are rare. If you have lots of steam engines with animated rods, you can slow a core2duo to a crawl eventually...

Yes, it happens now. But how do you know whether sprites need to be changed? To see that, you'd have to calculate the new sprite first, unless you implement the hooks I wrote about, limiting the utility again to the functionality provided by the hooks.

Am I right to assume that the animated rods aren't actually rendered each tick, but a frame from prerendered animation is displayed (the callback returns a sprite-number) ?

It's not a callback in NFO jargon, but yes, it returns the number of a pre-rendered sprite to use for the current vehicle state.

Anyway, how many games do you have that are actually that large?

With 37 ticks per second and 100 trains each having 9 wagons, you're already at 37.000 evaluations per second. And that's not a particularly large game, I regularly reach that in single player games. Besides, that's only the vehicle sprite evaluations, for stations, houses and industry tiles we also have callbacks that are potentially called for every tick and every tile that's redrawn.

I mean, what gain do you buy with the complicated nfo-language?

Versatility and performance. Because of the tight integration with the game's state engine, the only limit is the imagination of the grf coder, and performance is good because it's byte code that's heavily optimized for the usage patterns that you will need in displaying sprites. In principle, the byte code can even be compiled into machine code, at least for TTDPatch where we work with CPU instructions directly. It might be possible to do that for OTTD as well by adding the required assembly code for given platforms, though.

Saying, in essence, that "in two years the scripted approach will be fast enough" is not really a good argument...

Really, the only problem you seem to have with NFO code is that it's hard to write and even harder to read. There have been several proposals to solve this by writing a compiler from a high-level language to NFO, which is probably a more reachable goal than having to essentially rewrite all the features NFO provides but with a scripted backend instead of the NFO bytecode. Because even if you use LUA, you still have to code all the hooks it offers into the game, after all.

Don't reinvent the wheel just because you want to put a pretty hubcap on it.

[edit] It's 37 ticks per second, or 74 ticks per game day.

[BobDendry]

How fast is nfo btw ? It's run on a virtual machine, too, isn't it ?

I think we've found the winner to the "The Most Stupid Post of the Month" competition. You should keep your mouth shut on things you know nothing about.

Very useful post WhiteHand... Can you limit such quality posts to the off-topic section? Thank you.

Well, it was a hell of a lot more useful than A LOT of the posts present in here, with exception of the last page or two, which have actually shown some useful discussion.

[Expresso]

Well, actually the compiler idea seems quite nice to me, but make it in such a way that it uses human readable information as much as possible; that includes the ability to enumerate graphics used by the NFO files, so I don't have to memorize graphic numbers. Those enumerations could be resolved by the compiler, which could produce a NFO file and a linker information file (so you just link the graphics and NFO file together and don't have to figure out the correct order of the graphics).

In other words; I want to use my memory as little as possible when I'm codeing for a grf, the compiler and linker should figure the numbers and sprite order out for me.

[Brianetta]

Well, actually the compiler idea seems quite nice to me, but make it in such a way that it uses human readable information as much as possible...

You do realise that that is, um, the entire point of having a compiler?

[DaleStan]

LUA is a complete scripting language, which would seem like overkill to me. However, XML seems seems just fine for the job and provides a lower learning curve as LUA and a lot lower learning curve as NFO.

Besides the webbed sprite lookups, there are other problems with using XML. NFO is, among other things, Turing complete and self-modifying. I've not yet seen an XML spec that is either, unless what you're considering is merely a byte-for-byte translation of NFO.

If I'm wrong, please correct me, but I think the newgrf-thing is older than ottd. Obviously, the ttdpatch-developer don't have access to the source code, so they had to cope with what the executable offers and what can be hacked into it. The newgrf format is not as it is because it is clever or fast or has any other hidden benefits, but because the ttdpatch-executable understands it.

What Brianetta said.
Also, you're confusing the GRF container and the NFO language. We use the GRF container because TTD understands it. Support for NFO, however, is entirely found within the TTDPatch executable. Without Patch's help, a TTD executable wouldn't have the vaguest clue what to do with a newgrf file. It wouldn't even know to look for one.

As for performance, most of the nfo-processing seems to be done upon loading the newgrf, and that's effectively free time.

Most of the NFO processing is done at load, yes. But, as Patchman said, it is executed tens of thousands of times every tick. I've seen a TTDPatch game (so single player, 256x256 map) with over 11,000 vehicles. (Yes that's eleven thousand. There is not an extra character there.) This comes to just short of half a million vehicle sprite lookups alone, every single second, plus the vehicle callbacks, station and industry tile sprite selection and callbacks, industry production calculations, &c.. I would guess that NFO will start saturating a 1GHz processor at around 2 to 4 million lookups per second.
Unless your LUA scripts are at least half that fast, I really don't see the point.

The game then uses map<cargo_id,spriteset>::operator[] to lookup the correct spriteset (std::map is quite fast, O(log N), I think

It is my understanding that NFO is quite faster. Specifically, I'm pretty sure it's O(1).
Not to mention the minor detail of "OpenTTD is written in C, not C++".

That would solve some (most ?) problems, but it means stopping half way. You can still only do what newgrfs can do in ttdpatch.

Name one thing that NFO *can't* do. (Besides (see below) things that no one has yet considered doing.)

Lua could be allowed to do everything from the beginning.

No, it can't. You simply cannot write a language that supports things that no one has yet thought of doing.
If you want to claim otherwise, make sure your proposed spec includes a way to set train property 2B.

the compiler, which could produce a NFO file and a linker information file (so you just link the graphics and NFO file together and don't have to figure out the correct order of the graphics).

More nit-pickyness: An NFO file, by definition, already has the sprite references and the code properly interleaved.
If the sprite references are in some other file, or are not properly interleaved with the code, you're not looking at an NFO file.

[eis_os]

While I have to agree that NFO doesn't fit to well on some parts it showed until now a quite good base for us all to work with. Consindering the sets and features we provide with it. I have seen quite a few plans already of OTTD Folks, like ini files, xml, dats, did I forget something?

However currently I don't see a benefit for OTTD to switch to a other system without haveing a real killer feature.
The steady nfo base already created A LOT of nice graphics for TTDPatch, for most stuff it's quite suited. While not all are perfect and some features simple use a different system then the vehicle action321. Let's say newhouses action2, it still shows that the basic concept works.

If TTDPatchs byte based action language would be really so limited, I can't understand why the current versions of OTTD still don't support all stuff in it. An additional note, before someone should start to create a new language for new graphics, fix the spritenumber problem, because 11k sprites are a lot less. TTDPatch with 64k sprites already run ot of sprites...

-edit-
peter1138 told me he has already the necessary patch...

[Haukinger]

Besides the webbed sprite lookups, there are other problems with using XML. NFO is, among other things, Turing complete and self-modifying. I've not yet seen an XML spec that is either, unless what you're considering is merely a byte-for-byte translation of NFO.

Yep. Of course. Xml is a language for document-descriptions. You could exchange it with whatever document-language you imagine, even MS Word or LaTex or html. I thought of xml only because it is easy to read and write by hand and has good parsers readily available.

The game then uses map<cargo_id,spriteset>::operator[] to lookup the correct spriteset (std::map is quite fast, O(log N), I think

It is my understanding that NFO is quite faster. Specifically, I'm pretty sure it's O(1).

As said before, you're free to chose another datastructure, std::map is only used for illustration purposes. If I had to chose, I'd go for O(log N) and have a runtime-defined number of cargotypes instead of having to limit them at compiletime for constant lookup time.

Not to mention the minor detail of "OpenTTD is written in C, not C++".

That's no true. Have a look at the code, you can find some template-structs, and yapf uses template classes. Show me the C-compiler that compiles this.

That would solve some (most ?) problems, but it means stopping half way. You can still only do what newgrfs can do in ttdpatch.

Name one thing that NFO *can't* do. (Besides (see below) things that no one has yet considered doing.)

Lua could be allowed to do everything from the beginning.

No, it can't. You simply cannot write a language that supports things that no one has yet thought of doing.
If you want to claim otherwise, make sure your proposed spec includes a way to set train property 2B.

Wanna bet I can ? Just define a hook that calls a script-function that looks like this:

Code: Select all

do_vehicle( vehicleid )

Then expose the world-state to the lua-vm and call the hook every game-tick for every pure-lua vehicle. The script can then arbitrarily affect the game. The display-thread (or function) calls a hook like this:

Code: Select all

draw_vehicle( vehicleid, backbuffer )

for every frame drawn. There you have arbitrary behaviour and arbitrary graphics. Probably not as fast as pure C, but working.

[Haukinger]

However currently I don't see a benefit for OTTD to switch to a other system without haveing a real killer feature.

Agreed. The only real feature I see in lua is usability - for the newgrf-coder.

[peter1138]

Not to mention the minor detail of "OpenTTD is written in C, not C++".

That's no true. Have a look at the code, you can find some template-structs, and yapf uses template classes. Show me the C-compiler that compiles this.

Currently it's mostly C compiled as C++...

[Haukinger]

Don't reinvent the wheel just because you want to put a pretty hubcap on it.

Agreed. What about reinventing a wheel that's not welded to the car and requires a manufactoring plant to exchange it but one that can be replaced by the driver of the car using simple tools ?
Another thing for my better understanding - do you really have to update the sprites every gametick ? The positional information and cargos etc, ok, but the sprites ? Wouldn't it suffice to update them each time the game updates the screen ?

[belugas]

The only real feature I see in lua is usability - for the newgrf-coder.

So we are been asked to change the addition of objects language we are using (nfo) in order to have easyness for coding new graphics. Let's say it seems quite not a good reason, in my book. If the new language could provide something a lot more usefull then what nfo already does, it wold be interesting. But may I remind that never, since the idea of a replacment for nfo has been mentionned (and i'm not talking about this precise thread only), never have such a language been SHOWN with WORKING code?

It's easy to criticize. It is another to construct. NFO may be really hard to read/write, but it DOES the job been asked to do. You can think of NFO as ASM, if you wish. Assembler is hard to read ( i know,i scrutinize TTDPatch a LOT ) but it gives some incredible results. Same for NFO.

[Haukinger]

If you ask it to specify the looks and behaviour of objects in ottd, then, yes it does the job.

If I design a system with a scripting-interface I ask the scripting-interface to be easy to use in the first place. Otherwise the code could be in ottd or simply in a dll.