A grammar module to respect grammar rules when objects have a genre

Discuss translation of Daggerfall Unity and the required Daggerfall installation. Help other communities learn how to translate Daggerfall using any available tools and processes.
Post Reply
User avatar
Daneel53
Posts: 110
Joined: Wed Jul 15, 2020 11:15 pm
Contact:

A grammar module to respect grammar rules when objects have a genre

Post by Daneel53 »

Following a question on the discord Lysandus' Tomb, the question is: how is it possible to force Daggerfall to adapt texts to the genre of objects for languages where objects have a genre? English is a language with a very simple grammar: objects have no genre, adjectives don't need to be modified depending the genre nor the number, so English people that never learned another language have no idea of what I'm talking about. Let's take a very simple example.

In English, the sentence "the green %s" will be always valid, no matter by what %s will be replaced: a table, gauntlets... it always works fine.
Now if I take my language, French, and if I translate this string by "le %s vert", this is valid only if %s is an object of genre masculine and singular: "le chapeau vert", it's OK. But if the object is of genre feminine, it should be written "la table verte", the sentence "le table vert" is incorrect with two errors. And if %s is feminine and plural, "le bottes vert" should be written "les bottes vertes". Other languages as German or Polish are even worse because they have more than two genres and a more complex grammar.

Due to the simple English grammar, all games that are designed in English and that use variable expressions as %s where the word that will replace %s is unknown during the translation phase have no problem because there are few sentences where the genre may cause grammar issue. Only with humans and dual words as "his/her" or "king/queen", and for these few issues they invent a trick to choose one or the other word depending the genre of the concerned human. But they design nothing to be able to take into account the fact that objects have a genre in other languages. And so we, translators, are naked before this problem.

The only game I know where developers decided to implement ways to take this in account, probably because they are not native English speakers but Turkish, is Mount&Blade: Warband. They have implemented special expressions that permit to change the result depending the genre, at least for humans. For example, the expression {xxx/yyy} means "write xxx if the main character is male and yyy if she is a woman". So, a sentence in English as "Hello, you are beautiful today" said to the main character may be translated by "Bonjour, vous êtes {beau/belle} aujourd'hui", and the text parser of the game will write "vous êtes beau" or "vous êtes belle" depending the genre of the main character. This expression is the more simple one, TaleWorlds implemented in Warband other ones that permit to do things like that in other grammar cases, but not enough.
Their masterpiece in terms of respect to grammar is into M&B II:Bannerlord. In this game, they develop the concept of so-called "tokens", expressions written as {.xxx}, that are processed by a routine called ProcessToken. There are as much ProcessToken classes as languages implemented in their game, and each language has its own set of tokens and code to process them. The interest of the way it is done is that all the tokens are to be written into the translated files only and that the routine ProcessToken may be externalized in a dll for languages that are not natively translated by Bannerlord through the use of the library Harmony.

Before TaleWorlds launched officially the French language into release e1.70, the class FrenchTextProcessor that contained a void routine ProcessToken was already existing, so French translators as me that translated Bannerlord since its very first early access release created their own ProcessToken routine with tokens we invented. This lead to a dll called GrammaireFR that was able to implement French grammar rules needed to generate correct sentences into the French translation. And because this library, from which I am the last author, rely on different tokens that those used by TaleWorlds and has a totally original code, it may be used for other games. In other words, I have in hand the code of a French grammar library that could be used for DFU and any other game that would accept to call it.

The principle would be, in the game, to read a translated string from translation files, to store it into a string named for example StringToDisplay and to add a call ProcessToken(StringToDisplay) before to display it. The Grammar library would interpret the grammar tokens and send back into StringToDisplay a new string where the grammar tokens would have been replaced by a correct text.

To be more clear, let's take an example.

English string: "I want the green %s."
Direct French translation: "Je veux le %s vert." -> very bad if %s is feminine and/or plural: display of "Je veux le bottes vert." instead of "Je veux les bottes vertes."
French with tokens in translation files: Je veux {.le}%s vert{.es}
French processed by GrammaireFR: "Je veux le chapeau vert." ou "Je veux les bottes vertes.", OK!

The interest is that you don't have to implement in the game any specific code dedicated to the grammar rules of any language other than English, you just have to add one call to ProcessToken() with the string read into the translation file, and the grammar library will send back the correct string to be displayed. And if you do that call into the game routine dedicated to display string, you just have one line of code to add in the game and then you let the grammar library do the job before to display.

GrammaireFR.dll as it is today must be modified so that it becomes less dependent of Bannerlord content, but to transform it in a routine that may be used by several different games is something feasible without too much effort. In fact, when I developed GrammaireFR for Bannerlord, I was already thinking to a module that could be used by several games, but for that, devs for games that read translation files must accept to add one call into their game before displaying a translated string with grammar tokens. DFU may be the first.

You may watch a small video that shows the result of GrammaireFR upon the above example with that link:
https://drive.google.com/file/d/1PmZ6yu ... drive_link

I will provide later a small document that list all the tokens that exist into GrammaireFR and what they do.

No need to say that it could be possible to code grammar libraries for other languages, with their own tokens and rules, as it is done into Bannerlord that will stay a very good example for that. My code shall be public and translators in other languages may try to start from it to buid their own grammar module.

If you're interested, tell me what you think. :)
In charge of Project French Daggerfall and DaggerfallSetup, dev. of DFTools in English.
French translator for many Warband mods and Bannerlord.

User avatar
pango
Posts: 3359
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: A grammar module to respect grammar rules when objects have a genre

Post by pango »

Hi Daneel53,
We already discussed this topic elsewhere, so I'm glad you brought it here, fascinating topic!
Of course, as a developer, it raises a lot of questions...

How can the library know the gender of %s, and whether it's plural? Genders (and numbers) are language-specific (for things, at least), so it cannot be injected by the game, so I assume the library has to contain some dictionary to infer that, that need to handle all the nouns the game can throw at you.
And then, if they're more than one word parameter, how can you tell with what word each token should agree with? Are there proximity heuristics, or a syntax to explicitly tell that (say {s1.es} to adjust with the gender of s1, etc.)? What if you have to agree with several nouns (s1=pantalon s2=veste "Un{.e}{s1}et un[.e} {s2} vert{s1,s2.e}")? Etc.
I'd be really curious to see what the API looks like.

On a related topic, I recently read in the AMA thread created by Julian Lefay on Reddit that this is a pet peeve topic of him, that he learned several languages in different countries, and that he worked on a Daggerfall translation subsystem that were sadly never released (here). Ah!
Mastodon: @pango@fosstodon.org
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

User avatar
Daneel53
Posts: 110
Joined: Wed Jul 15, 2020 11:15 pm
Contact:

Re: A grammar module to respect grammar rules when objects have a genre

Post by Daneel53 »

Hello pango,

Here is a link to take a document that explains what can do GrammaireFR, what it cannot do and what it could do in the future :
https://bit.ly/3VRdAt2

This answer to some of your questions, but not all. This conversation is to be continued... :)

Note: Sorry for non French speaking people, this file mainly comes from the Readme file of GrammaireFR.dll v2.0 as it was released three years ago. It was done for French translators, it talks about French grammar, so it is written in French. If I have time, I will try to realize an English translation of the document, but since then, DeepL is your friend. ;)
In charge of Project French Daggerfall and DaggerfallSetup, dev. of DFTools in English.
French translator for many Warband mods and Bannerlord.

User avatar
Daneel53
Posts: 110
Joined: Wed Jul 15, 2020 11:15 pm
Contact:

Re: A grammar module to respect grammar rules when objects have a genre

Post by Daneel53 »

I realized that GrammaireFR, as it was developed three years ago, is much more linked to Bannerlord that I tought. Obviouly I will have to change the logic of how it is done and I will have to create additional tokens.

I am currently rewriting it so that it becomes a French grammar library that can be easily used by any game and be usable with few modification by already existing games as DFU. During the process, I will switch the comments in code in English so that the code may be modified by other to build their own language grammar library if there is other fools ready for that.

So even if I don't give regular news here, I'm working on it. I should have something in hand in two or three weeks.

See you later, alligator... ;)
In charge of Project French Daggerfall and DaggerfallSetup, dev. of DFTools in English.
French translator for many Warband mods and Bannerlord.

User avatar
pango
Posts: 3359
Joined: Wed Jul 18, 2018 6:14 pm
Location: France
Contact:

Re: A grammar module to respect grammar rules when objects have a genre

Post by pango »

I was still digesting the documentation, but one interesting point I read was that at times the library could need hints from the game to do a perfect job; Like getting hints about whether some character is male or female, something that cannot always be correctly guessed from the name alone.
Given we have control over the game engine, that's certainly something that could be done. Since the list of hints localization libraries may need is not closed, I was thinking about sending some kind of "context" dictionary, something similar to a JSON object (even if the type is not JSON-related):

Code: Select all

{"Bedastyr Woodhouse":{"gender":"male", "count":1},"Vannayne Mooring":{"gender":"female","count":1}}
Mastodon: @pango@fosstodon.org
When a measure becomes a target, it ceases to be a good measure.
-- Charles Goodhart

User avatar
Daneel53
Posts: 110
Joined: Wed Jul 15, 2020 11:15 pm
Contact:

Re: A grammar module to respect grammar rules when objects have a genre

Post by Daneel53 »

Hello guys!

A fast word just to say that the French grammar routine is working fine. I've spend many time to search into the code of DFU, on which I have no knowledge, where I have to place calls to the ProcessGrammar(text) routine in order to cover all the screens where text is displayed (inventory, history, quests, conversation screen...).I had to modify five scripts (*.cs), seems that it's complete today.

I spend an even longer time to find how to technically implement this in DFU so that:
1 - DFU shall keep on working and compiling even if there is no non English grammar module connected.
2 - It shall be easy to connect a grammar module, whether it is the French one or another one if somebody is fool enough to try to do the same for their own language.

It is quite long because, being an old man, object oriented programming is not part of my programming history (I come from Basic, Fortran, Pascal and C, scratched a little Java and C++), so I struggle to understand how to use correctly things as namespace, abstract, virtual, public, private, static, override, etc. I think that today I've understood these concepts and found a way to implement all this, but I have still one step to climb: enclose my French grammar routines into a mod that can take the hand when activated and connect the French gammar routines to the game. Once again, if the French module is not present, the game will work as before.

Once all these technical goals shall be achieved, I will submit to the people in charge of DFU the needed modifications of the DFU code that must be done, so that it becones possible to connect a grammar module to DFU. DFU release 1.1.1 is out, I hope that all this shall be present into a future release 1.2.0.
In charge of Project French Daggerfall and DaggerfallSetup, dev. of DFTools in English.
French translator for many Warband mods and Bannerlord.

User avatar
Daneel53
Posts: 110
Joined: Wed Jul 15, 2020 11:15 pm
Contact:

Re: A grammar module to respect grammar rules when objects have a genre

Post by Daneel53 »

OK, it's done!

The DFU French translation already has a mod that is needed to translate the names of shops in town that are generated each time a new game is launched. I just added the script that contain the French grammar processor to the scripts of the mod, added one line into the Awake() method so that the French grammar processor replace the default one I've created (a stub that does nothing)... and it works. :)

If the French mod is disabled, the default processor is used and DFU works as usual. If it enabled, the French grammar processor is called and the grammar tokens (inserted into Internal_RSC.csv for example) are processed. Fine!

Next step: write a doc to explain to DFU dev team how DFU code should be updated in order to permit a grammar mod to be used into the game. I don't have many free time nowadays, will try to do that for the end of the month. Due to the nature of the modification to be done, I cannot just submit pull requests into the game github. I must first ask the team if they accept to integrate such a functionality and explain how it will work.
In charge of Project French Daggerfall and DaggerfallSetup, dev. of DFTools in English.
French translator for many Warband mods and Bannerlord.

User avatar
Daneel53
Posts: 110
Joined: Wed Jul 15, 2020 11:15 pm
Contact:

Re: A grammar module to respect grammar rules when objects have a genre

Post by Daneel53 »

Done also!

I've wrote a document that explain what is to be done in DFU so that grammar processors may be used, and I've posted a message on the Daggerfall Unity github into the entry DFU 1.2 in the Issues.

I've no problem to make this document public, you will find it hereunder. 8-)
Attachments
Grammar processors for DFU.zip
(135.3 KiB) Downloaded 49 times
In charge of Project French Daggerfall and DaggerfallSetup, dev. of DFTools in English.
French translator for many Warband mods and Bannerlord.

Post Reply