DjUnique's Arcanum modding tools update

Long time, no post.

For those who don't know me: I am the creator of a few Arcanum modding tools such as ScriptEd, DatBuilder and Dialog Debugger back in 2001-2004.

The reason why I decided to post here now is because I have lost the source code to all my Arcanum modding tools due to some unfortunate events (Linux installation, accidental formatting of the partition where all my backups were stored, deeply regret being so hasty) so I have to start working on them completely from scratch in order to get any updated versions released. This is just to let people know that these tools are not dead as far their development is concerned even though I may have stated somewhere that the tools are discontinued. I have tried recovering the deleted files from my USB hard drive but it's no use, most of the important files come up corrupted. It doesn't matter so much anymore, seeing as most of the code was a horrible spaghetti mess that even I probably couldn't follow myself.

The first project I've taken up on is the script loader/compiler unit which will also become a part of a forthcoming ScriptEd 1.50 release. This time around the text parser should be a lot faster and less likely to get confused by the input. The reason the old code is so slow is because it spends needless amounts of times looking up stuff and iterating through lines a few too many times. From what I've been testing with the new unit so far, everything seems to work perfectly. It took me about 12 hours in total to write the script loader which is definitely less time than the previous one.

I remember the old script parser used to choke on a lot of statements (such as lines containing a PC Variable, that's no longer a problem) and produce incorrect output in some cases. The GUI will be completely new but the text syntax is 100% compatible with older versions of ScriptEd.

The ScriptEd tool itself also seems to have many bugs that I'd like to address in the next version so there's more than enough reasons to go for this.

I'll post more information sooner or later when things progress further. Starting work on the new GUI now.

 

I'm a really big fan of your work, especially ScriptEd. If it weren't for ScriptEd, I would probably have stopped modding Arcanum about 6 months ago. I also use DatBuilder, although not nearly as often. I am currently using wordpad for dialog files, but I'd be willing to try a new release of Dialog Debugger.

I have a lot of comments on what lines don't parse quite right, but since it sounds like you are starting over, I'm not sure they will be helpful.

I'm sure that virtually everyone who mods uses ScriptEd at least 90% of the time, and if they don't, then I think they should. It is difficult to overstate how important this is to creating and editing new scripts.

 

For the lines that don't compile correctly, just list them here and I'll test what the new parser does to them.

So far I've tested the compiler with a 107 line script from the original Arcanum module (01000Praetor.scr) and the result was 100% accurate so I don't think there's anything this new parser can't handle. And if a certain line doesn't compile correctly, it's much easier to fix now.

Here's an early development screenshot of ScriptEd: http://www.djunique.org/files/devshot1.jpg

Not all menus are implemented yet (Plugins and the module selector for example) but I'll get there eventually.

 

A little update.. I've been testing the performance of the compiler and it is really fast. Only when the line count goes above 1000 it starts to show a bit of slowing down. A 1000 line script takes about 0.5 seconds to compile, 2000 lines takes 2.11 seconds, 3000 lines bumps the time up to 4.9 seconds and so on. As an additional extreme test, I tried a huge 10000 line script and it took 59 seconds to compile. However, since you never need that many lines in your scripts, the compile times are kept pretty much in fractions of a second. The compile speed depends heavily on the processor used. These tests were performed on an AMD 64 X2 4600+ processor.

So far new to ScriptEd 1.50 will be:

- Plugin support
- Unlimited number of lines in scripts
- Helper scripts via a Pascal-style scripting engine (used for script templates and other things)
- Built-in dialogue editor - if a dlg file matching the name of the script is found, it will be automatically loaded. The dialogue loader parses the dialogue tree into nodes (npc text + player responses or just npc text if it's a float line) This is what the dialogue editor will look like (a very rough draft): http://www.djunique.org/files/devshot2.jpg
The add/delete buttons for nodes and player options will be added sooner or later.
- Module data loading - when a module is selected, most common mes files in the mod folder will be loaded into memory and can be accessed via the main menu

+ many other things I haven't yet thought of

More news later as work progresses.

 

Lookin' good! You're going to make a lot of people happy with this.

 

Wow. Wow. I am pretty excited about this! I never expected to ever see another update for scripted. The new screenshots look good, and the interface for the dlg editor seems like a particularly good idea.

I really like the idea of nodes here. The 'use text from male npc line' is a great idea too. I assume that there is going to be condition and result fields for the PC lines, so I won't comment on that yet. Am I right in thinking that this means that I won't ever have to renumber PC lines? What happens when there are so many PC lines in a section that the next NPC line needs to be renumbered? Also, I'd suggest letting the user name the nodes, like "greeting line" or "greeting line - has met PC before" so the user can keep track of them. Also, is there an option to have the goto line field for the PC dlg track a node rather than a line number (in case NPC lines get renumbered)? I imagine that this might be difficult or time-consuming to create, but it might be a suggestion for later versions.

The scripted speed sounds ridiculously fast. The old scriped wasn't particularly slow, but this is still a big improvement.

The only feature I really found lacking from scripted (besides a few problems with compiling) was that scripted wouldn't renumber the goto lines, which sock monkey could. These 2 things are the only reasons I ever used sock monkey, and I imagine that the compiling issue will go away soon. Perhaps scripted might one day use nodes as well? Or what's more practical, just include an option when compiling to renumber goto actions as well, and have each goto line increase by the same amount as the line number, etc. It wouldn't be a good idea to have this on for all compiling, just once in awhile.

I have to say that overall, i'm impressed by how quickly this has progressed. Even if you didn't look at any of my suggestions, I doubt that I would be any less excited about these releases.

 

I'd say at this point all suggestions will be considered for inclusion. Having worked on a Fallout 2 Dialogue editor for about 3 years (with long pauses in between) I don't think there's anything I wouldn't be able to implement. The first beta version of the application is starting to be near finished, I'm currently implementing some neat script writing features that should speed up manual script creation a lot. These include autocorrection (for example when you type 'rets' it will automatically become 'return and SKIP default' etc - these will be placed in a text file so more entries can be added later), automatic script compiling when you press enter, completion proposal listiing all available commands and so forth. I will definitely look into implementing goto line remapping already in the beta release since it will really come in handy. The other suggestions are not bad at all really. Node naming is definitely doable and since the dialogue loader preserves comments found in the dialogue file, I could store node names with a special formatting in the comments (such as '// NODE: StartNode' or something like that) before each npc line so that they are automatically saved and loaded.

I am planning to include a line number remapping tool in the dialogue editor which goes through the dialogue and changes the start lines of each node with a given increment between nodes (like 1 - 20 - 40 - 60 etc with player option lines going 2,3,4 - 21,22,23 etc). Player option line numbers are remapped accordingly. Of course I will also try to make the script update its line numbers in each dialogue / float line call (since the script also exists in binary format most of the time it'll be easy to iterate through the script to check for dialog or float line opcodes and get the line number from the parameters)

Also, I've already decided to use the GPL license for ScriptEd meaning that the source code will also be available upon the initial release. I've kept the amount of extra components as small as possible (currently JVCL, JCL, DelphiWebScript and SynEdit) so that compiling the source would't be too much of a hassle. The app is completely written in Delphi 7.

No estimate on release date but I'll try to get things done as soon as possible. The rest of the development cycle will pretty much depend on user feedback. I'll set up a Mantis project entry for ScriptEd on my modding tools so that people can report bugs and other things (will be much easier to keep track of all issues that way)

 

Here's some of the lines that didn't compile right in scripted:

1. IF global flag gv 1 is set
THEN do nothing
2. IF local flag lv 1 is set
THEN dn

-only accept numbers for (num)

3. call script gv 1 at line gv 1 with triggerer Triggerer and attachee Triggerer at second gv 1

-(num) value 2 and 3 only accept numbers

4. call script attached to clo at point gv 1 at line gv 1 with triggerer triggerer

-(num) value 1 and 2 only accept numbers

5. remove from time queue the call to script 1 with attachee attachee

-never works

6. toggle attachee open/closed

-I'm not sure if I created the opcode macro for "clo"="Current Looped Object" or if it came with scripted, but it gets confused here.


All in all, not that bad, because I doubt that many other people use so many global variables in "call script" actions.

 

All of the above lines compiled 100% perfectly with the new version with no problems at all.

Working on script line remapping now, it's a real pain to figure out but I'm almost getting there. This will increase development time a bit, really, but I do want to provide the best damn script editor ever created.

 

Success! ScriptEd can now track changes to goto commands. The changes are tracked script-wide so no matter how many script lines you remove or add, the reference should stay in the correct line, unless something goes wrong of course. I've spent a couple of hours almost tearing my hair off trying to get this routine to work, fortunately I did manage to get it working before I lost my sanity. I still have some access violation errors to weed out but so far it's working quite well.

The way this tracking is done is as follows:

Before each compile, the program scans through the current script data in memory and adds all lines that have the goto command to a script reference table. Then after compile it scans through the list and checks the lines that these entries link to, if changes are found, the program starts scanning from the linked line to the end of the script to see where the link actually is and changes the references if needed before the script is decompiled back to the editor. (It actually does this - decompiling the compiled lines - to maintain formatting)

A simple example would be like this:

Code:

1. IF local flag 1 is set
   THEN goto line 4

2. dialog 1

3. return and SKIP default

4. dialog 21

Now we add a line to the position where the dialog line call resides

Code:

1. IF local flag 1 is set
   THEN goto line 4

2. dialog 1

3. return and SKIP default

4. float line 1 above Attachee

5. dialog 21

After pressing compile, the result will be:

Code:

1. IF local flag 1 is set
   THEN goto line 5

2. dialog 1

3. return and SKIP default

4. float line 1 above Attachee

5. dialog 21

A little note though: It will NOT track goto calls that refer to lines out of range (more than the current line count) because that will raise an error (we would be attempting to read a variable that hasn't even been allocated)

 

New update...

I've been actively working on the integrated dialogue editor. Most basic functions are in place such as adding, inserting and deleting nodes. Player option editor is almost done, still need to enable actual editing of data. It has a few neat features, such as a visual display of a script line call (meaning you can see the actual line the dialogue is calling). The following screenshot will probably give a much better idea what I'm talking about:

http://www.djunique.org/files/devshot3.jpg

This screenshot shows how linking to a script line will be much easier now, no need to peek at the script separately when you can see them directly in the player option editor.

Links are done on to node names, rather than dialogue numbers so they will always resolve to the correct line number if they get remapped.

The dialogue editor window looks like this now:

http://www.djunique.org/files/devshot4.jpg

The 'Remap Line Numbers' button performs a renumbering operation on the dialogue, also taking into account references to each node in the script and in the dialogue. The following references are being remapped currently:

- Script opcodes 'dialog (num)', 'float line (num) above (obj)', 'print line (num) with a message class of (num)'

- regular dialogue links

- fl NUM - dialogue result codes

However, one of the things I still need to figure out is that when a dialogue line number is stored in a global or local variable, how is it going to know if that particular number actually refers to a dialog line or something else.

The 'Edit Dialogue Header' button opens a simple window with a memo control to write a message to the beginning of the dialogue. When a dialogue file is parsed, the comments from the very first line of the dialogue will appear here.

Also node names are stored in the dialogue lines in comments in the format '// NODE: <nodename>' so that whenever the dialogue is being edited with ScriptEd, it will always be able to import the node names correctly (even if there are comments after the node name specifier).

I still can't give an estimate when the first stable version is ready but I'm willing to put up an early beta installer if there are any brave souls wishing to test this program and give feedback on what to add and so on. Would help me a great deal in developing the tool further.

 

Why so late... I had to make 80% of my mod using Monkey... Toikas tool have really bad GUI but work correct

 

You could have used earlier versions of ScriptEd. Actually, considering that your mod is done (it is, right? just not translated into English?), I'm surprised you're still alive after having used sock monkey so much. In fact, despite the fact that the new version may come out soon, I'd still get the old one right now if I were you.

DJ_Unique - I would love to help with bug testing and whatnot. Those screenshots didn't load for me, but this sounds pretty awesome. It looks like you incorporated even some of the most difficult ideas I had. Can't wait to see it! Hopefully this will help with the whole desire to bang one's head against the wall condition sometimes caused by modding Arcanum.

 

My domain seems to have crapped out, it gets redirected to a link farm which is probably a sign that some bastards have hijacked it. WHOIS information still reports the correct information.

Here are temporary fixed links:

http://de01.rackglobal.com/~djunique/files/devshot3.jpg

http://de01.rackglobal.com/~djunique/files/devshot4.jpg

 

I tryied use earlier versions, but I rember that I had two or three situations that I coudnt make correct script using yours tool because of somme missing options in values. And you have to know that my scripts are really strange and huge... Specialy when I have made my own system of drugs and addict and bad and good effect of it.

 

MackaN - same here, man. I still found that 98% of my scripts worked, so I just use sock monkey to fix the parts that got compiled wrong. Since sock monkey renumbers lines and goto actions, it's not that hard.

I use this action a lot:

0. call script attached to Current Looped Object at point Global Variable 2 at line Global Variable 1 with triggerer Triggerer

Still, ScriptEd correctly compiles other lines like this one, which are also not that widely used (and I suspect only someone like you or I would use it, although there are a few Arcanum scripts that do):

0. Dialog Local 0

Dj_Unique - that looks sick, man. Is renumbering going to parse the whole rest of the file if you add in a new node in the middle of a script or dialog? Also, as to evaluating the Dialog (num) action where (num) is a variable not a number, is it possible to just block off those lines? If the user has some kind of complicated math system to determine which dialog line the script calls, the user should probably be the one to keep track of that value if lines are significantly renumbered. It's why I space my dialog lines apart by values of 10, 20 or 100.

One minor comment on the Dialog Editor: since most dialogs use either -4 or 5 for the intelligence check, it might be easier to use a check box than to have to always type it in, as a shortcut kind of thing. For example:

[ ] -4 [X] 5 [ ] Other: _____

And a check box for gender might work too, with "both" being assumed if no boxes are checked:

[ ] Male Only [ ] Female Only

I hope these suggestions don't slow down your development process much. They might be ideas you consider after you have a stable version.

Thanks again for the hard work!

 

Well, I am happy to say that the lines you mentioned above will have no trouble compiling in the new ScriptEd.

I actually added a variable assignment remapping feature yesterday. When it encounters the opcode '(num) = (num)' it will check to see if the second parameter really is a dialogue line number and if true, it will change the value to the new remapped line number.

There is a switch to turn this feature off in the option window that comes up when 'Remap line numbers' is pressed.

The remapping process is currently performed manually, it's not automatic and I'm not sure if I should make it that way. (Wouldn't be difficult since I could just add a call to the code on the line number remapping button whenever a node is added)

The remapper also adds comments to the script (yes, comments are supported, saved as a separate .cmt file that is only used by ScriptEd) noting what was changed on each line and what the old value was.

As for the player option editor suggestions, I'm adding them as this message is being posted.

Additionally, I had plans of creating a complete interface for dialogue conditions and actions (lots of work, will probably implement it in a later release) but for now these conditions and actions have to be entered manually.

I'm now prepping the first rough draft beta version for release. It still lacks some features (such as an add script line wizard for teh n00bs which I will try to add to by the time the stable version is ready, also not too many preferences items are present) but so far from what I've tested it myself, it's pretty solid and it should already be possible to easily create new scripts and dialogues with this beta version. I'll post a link later when the time comes.

 

That comment on script is a really good feature, one I would have suggested but I figured it might be asking a lot to create a new file/database to hold them. It does look like keeping track of nodes will ease some of that problem, however, as I would mostly want to comment on what part of a script is called by the dlg and why, or from a completely different script, etc, which will be less complicated if they're already organized.

Are normal comments viewable with dialog editor? I have some comments embedded already, whether above the NPC line or to the right of PC lines. It would be nice to have a little text box in the dialog editor to see some more extended comments. A comments section of the dialog as a whole would be nice as well - I usually keep track of things like local flags and counters, especially for generic dialogs where I set some of the flags on the actual NPC, not just in the script.

When might we expect to see the earliest release of a beta?

 

Comment parsing for the dialogue is fully working at the moment. For dialogues that are saved by ScriptEd, node names and their descriptions are parsed from comments that contain the keywords "NODE:" and "DESCRIPTION:". The description field can contain multiple lines. If the node does not contain either of these keywords, the comments are assumed to go to the node description field. This is for the non-ScriptEd saved dialogues to maintain as much original comments as possible.

The 'dialogue header' that goes to the beginning of the dialogue file can be used as a global comment on the dialogue.

Currently I have a problem with the dialogue header because it has no way of knowing what belongs to the header and what belongs to the node so all comments in the first node will end up in the header. This means that the comments that are supposed to be in the first node will have to be copypasted from the header to the node description. I hope this isn't too much of trouble because it would speed up the release of the beta if I didn't have to struggle with this issue right now. It's only happening with dialogue files that were not saved by ScriptEd. Once the dialogue is re-saved with ScriptEd, the header will be completely separate from node comments.

I'm thinking it will be either today or tomorrow. I'll post the link into a separate thread for clarity's sake, then we can continue there with all the feature suggestions and other things.

 

For a first release, the header field might be a minor problem.

Could you parse the header before nodes so everything before node 1 is header? Or, rather, parse the nodes but then parse the header before the comments. missing a comment on node 1 is probably less important than having a gigantic comment on node 1 that is 10 or 20 lines long (meant to be the dlg header).