DOS/4GW Port of FTE

March 31, 2019 dos code

A long time back, hidden away in a much larger post, I indicated that I wanted to port the FTE text editor to build against the Watcom toolchain so that it ran under DOS/4GW instead of CWSDPMI which the current (old) DOS build built with DJGPP uses. The reasons for this are that FTE includes menu options for running external tools (such as compilers, debuggers, etc) but because it runs under CWSDPMI, if you try to run most things that use DOS/4GW, they will crash. Different DOS extenders shouldn't be mixed.

I finally did this today. Hooray! Well, I'm probably the only person out there who cares about this, but still, hooray!

I've uploaded the changes to a Github project which includes a binary release for anyone else who may be interested in such a thing.

In addition to being build with the Watcom toolchain, I also took the opportunity to fix the broken / un-implemented DOS support for different sized screen text modes, such as 80x50 which I consider mandatory for a text editor under DOS these days. The non-platform-specific bits of code was already set up to work with a variety of different screen sizes. However, the DOS-specific bits that existed in the project (which was intended for DJGPP) did not include support for different DOS text modes via interrupt 10h. A rather curious omission since there was a non-trivial amount of other DOS VGA work that had to be done in the code to get it running under DOS in the first place by whoever did the DJGPP port way back when. Perhaps that person only used 80x25 and decided to just leave it at that. I suppose I'll never know. Meh.

Tools Development and The Need For GUI Code

December 24, 2018 dos code

As some of the "foundational" pieces of code have been coming together, I've realized that I need to get some tools created to assist me in content creation. This would be for things from asset conversion to tilemap editors.

Asset conversion tools (e.g. going from BMP/PCX to a game-engine specific format optimized for faster loading) can really just be command-line driven which makes it simple enough to create. For more involved things such as a tilemap editor, this requires some GUI code.

I made a few tools in straight DOS using QBasic back in 1998/1999 or so with my own homebrew GUI code that was pretty terrible as I recall. But it was my very first experience writing such code (and only a few years after I first learned to program), so perhaps I should not be too hard on myself. I actually want to add a "projects" section to this website in the near future where I can post that old code just for fun, but currently I don't have it in a presentable format to share for this post.

Very soon after, I moved to Visual Basic (first v4, and then v6) and of course, that made writing such tools infinitely easier. I was actually pretty happy with the tilemap editor I wrote (from scratch entirely in a weekend):

For my current DOS projects, I once again need to revisit GUI code 20 years later. There are some libraries I could use of course, but I am going the "Do It Yourself" route (a.k.a. the "Not Invented Here Syndrome" route) for fun and for the (re-)learning experience. With that in mind, I decided I wanted to pursue an Immediate Mode GUI implementation. Probably the most well known implementation of this these days is the Dear ImGui library.

I began reading up on it in more detail, having not implemented such a system before in any project I'd worked on. This video in particular is a great introduction to it (and I believe, the original video on it) if you're not too familiar with why this method of implementing a GUI might be useful and where/when. As well, I found this tutorial series to be quite useful.

Eventually I did get something working myself:

It's far from perfect, but I'm happy with how it's coming along.

while not KeyState[KeyEsc] do begin
    event := PollEvents;
    if event <> nil then UIProcessEvent(event);

    Cls(9);
    UIBeginFrame;

    UIBeginMenuBar;
    if UIBeginMenu('File', 100) then begin
        UIMenuItem('New');
        UIMenuItem('Open');
        UIMenuItem('Save');
        UIMenuItem('Save As');
        UIMenuItem('Quit');
        UIEndMenu;
    end;
    if UIBeginMenu('Edit', 100) then begin
        UIMenuItem('Cut');
        UIMenuItem('Copy');
        UIMenuItem('Past');
        UIMenuItem('Clear');
    end;
    if UIBeginMenu('Help', 100) then begin
        UIMenuItem('About');
    end;
    UIEndMenuBar;

    UIBeginWindow(50, 30, 250, 180, 'window!');

    if UIButton('button 1') then
        clicked := true;
    if UIButton('button 2') then
        break;

    UISetNextID('vertSlider');
    UIVertScrollBar('', 20, 0, 2, value);
    UIHorizScrollBar('slider!', 0, 0, 200, value2);

    UICheckBox('checkbox', checked);
    UIListBox('list!', 0, 3, list, listScrollY, listSelection);

    if UIRadioButton('one', (radio = 1)) then radio := 1;
    if UIRadioButton('two', (radio = 2)) then radio := 2;
    if UIRadioButton('three', (radio = 3)) then radio := 3;

    UIEndWindow;

    UIEndFrame;

    DrawMouseCursor;
    WaitForVsync;
    Flip(BackbufferLayer);
end;

Some stuff doesn't work completely yet (such as the menus). As well, I'm missing two main widget types: textboxes and dropdown lists. Actually, I'm not expecting textbox widgets to be too difficult to implement. In fact, the input events system that I originally added to libDGL was originally added purely because I knew I was going to need such a thing to more easily implement such a textbox widget in the future!

However, I am relatively happy that things like automatic layout/positioning mostly works well, as well as automatic widget ID generation. Well, except that, as you can see above, there is an escape hatch for this in the form of UISetNextID that can be used when/if needed.

The idea with pretty much all of the widgets is that the widget function (e.g. UIButton, UICheckBox, etc) takes simple properties like a label string and maybe a couple other common style- or functionality-related properties as well as state variable(s) which are passed by pointer/reference to the widget. If the widget state changes, the variable(s) are updated immediately with the new state and the widget function will return a boolean to indicate that the state was changed. This boolean indicating that the "state was changed" is a bit context-dependant. For example, for a UIButton, it doesn't take in any explicit state variable itself (that is managed by the UI system in the form of widget IDs being tracked for what widget the mouse is currently hovering over, etc). But, the UIButton still returns true/false to indicate if it was clicked or not. For a UICheckBox which does require an explicit state variable to be passed to it, it returns true/false to indicate that that state variable was changed in response to a user action. This allows the application code to easily respond to the most common user interaction events for widgets with just simple if logic. Which is really, really nice actually! The previously linked video on immediate mode GUIs explains this idea really well and why it's nice for programs which do real-time rendering.

The key thing for me with this GUI code is that I do not want to create a totally generic UI framework. I only need to create something that will allow me to create some basic tools easily enough. Maybe eventually I'll want to add support for things like custom themes so that this could be used directly in an actual game project while matching the game's look and feel. Or, later on I might need more widget types. The list goes on, but I do not want to get lost in a never ending quest to get a "perfect" UI framework up and running, because well, it really would be a never ending quest and frankly, I don't need such a thing!

Right now I think I'm still going a bit too far with this by keeping support for things like tabbing between widgets. Will I actually use this feature in tools I create for myself? That remains to be seen (I suspect not, but you never know).

Some of the widgets leak implementation concerns a bit here and there which I'm not happy with. I've decided to just roll with it for now and hope that inspiration will kick in later and I'll be able to clean up these warts later on.

For example, the UIListBox widget requires two UI state variables to be provided to it. One of them, listSelection, is obviously needed (it holds the user's currently selection). The other one, listScrollY, holds the list's internal scrollbar scroll position and is kind of silly, but I didn't have any clever ideas for how to efficiently and automatically manage state of internal child widgets (the UIListBox widget internally uses a UIVertScrollBar widget).

Another example of this is in the menu code (which is mostly unimplemented as I write this). UIBeginMenu takes a second parameter which is the horizontal width of the menu to be created. This is kind of unfortunate, but I'm not exactly sure as of yet how to have this automatically figured out in a way that doesn't involve some kind of deferred rendering approach. I'm currently against the deferred rendering approach as I don't want to have to keep that kind of state around in memory somewhere... but it might be the best approach as it also solves another problem with my current approach which is that overlapping sub-menus would not render correctly. I might leave deferred rendering as a "v2.0" feature, heh.

Anyway, I'm hoping to have the UI stuff finalized in a workable state (even if it's not perfect) within the next week which will allow me to move onto creating tools for myself. Additionally, this UI code will need to be ported back to C so I can use it with libDGL eventually. I've been doing it with Turbo Pascal mainly for the reasons mentioned in my last post ... I find Turbo Pascal to be quite a bit faster for prototyping and experimentation.

Turbo Pascal

October 14, 2018 dos code

Turbo Pascal is one of those development tools from the late 80's / early 90's that I had heard of but never actually used or even seen. In fact, the Pascal programming language as a whole I can describe the same way as it pertains to myself. Until 2018.

I actually now greatly regret that my younger self in the mid 90's did not get a chance to use Turbo Pascal at that time. As I've written about before, I started out with QBasic and then moved on to C/C++, and then soon after went back to QuickBASIC 4.5 while adding assembly into the mix to do more performant graphics. The reason I ended up moving back to QuickBASIC from C/C++ was that I greatly preferred the easier/quicker edit-compile-run cycle and the simpler debugging environment (plus the inline help system in QBasic/QuickBASIC is really awesome too).

That last point is what really makes me, in 2018 looking at Turbo Pascal for the first time, think to myself: "Wow! This is what I really would've loved at that time!"

I've actually had this since early this year but only really dug into it over the summer. I originally got it after reading posts on some vintage/retro computing forums with a number of people praising Turbo Pascal as a great development tool mixing a great IDE, ease of use, inline help and advanced features (pointers, inline assembly, etc). I was intrigued and I figured that maybe it might be good to play with a bit, as I had been interested in getting into FreePascal and Lazarus a bit anyway (maybe, for various reasons which I could probably write a whole post about).

So what do I like about Turbo Pascal and why do I think my younger self would've really liked it as well? In no particular order:

  • Easy to learn language with (to me anyway) a kind of "BASIC-like" feel to it.
  • Really awesome modules/units system. This honestly feels like it was way ahead of anything else at the time. It makes code re-use between projects incredibly simple. Just copy the TPU file (compiled unit) around and reference it in your program under the uses directive to start using it. Really easy.
  • Pointers!
  • Inline assembly (though, 16-bit / 286 only). Huge deal for me, as I remember getting annoyed at debugging assembly issues in my QuickBASIC/assembly mixed language projects. The 16-bit only limitation can be worked around in a hacky way by directly writing hex opcodes into your inline assembly if needed. It amuses me that I've actually heard Turbo Pascal referred to as a "shell for assembly", referring to the fact that a lot of projects where speed really mattered would have large amounts of inline assembly everywhere. For me in 2018, if I ever work on a project that gets to that point, I'd likely just switch it over to C honestly.
  • Slightly more advanced type system than BASIC, but still quite simple. One thing I've noticed so far is that I do generally feel a bit safer writing Pascal code that uses pointers (for example) then I do with C, I think due mostly to the type system.
  • Very fast compiler! And your code is actually running compiled (even from the IDE) rather then being interpreted, as in the case of QBasic. So your code is going to be a fair bit faster then QBasic right away. One side-effect I've noticed as a result of the blazing fast compiler is that I'll often compile my code as I write it (even if I don't intend to run it quite yet) simply as a check for any syntax mistakes, etc.
  • Beginner-friendly IDE that is very fast and that allows you to immediately start writing code and running it (exactly like QBasic). Also includes debugging support that is roughly on par with QuickBASIC (but does have some extras, like inspecting register values while stepping through your inline assembly).
  • Syntax coloured highlighting in the IDE!
  • Inline help system with search and plenty of examples.
  • Run-time checks to help you debug code (which can all be turned off for additional performance).
  • The IDE runs itself under DPMI (optionally) so that your code (which always runs in real-mode) has access to all of the remaining 640k conventional memory. This is a massive improvement over QuickBASIC! I very vividly recall getting really frustrated with one of my late QuickBASIC projects which was becoming impossible to run from the IDE due to it always sucking up 200-300k of conventional memory.

I've often read from people who learnt to program as kids in the 90's that they progressed from BASIC to Pascal and then to C/C++. I can kind of see now why that progression makes sense. To me, Turbo Pascal really does feel like either a much more advanced QBasic, or a kind of C/C++ "on training wheels" (sort of).

Turbo Pascal 7 also includes some object-oriented programming features. Actually, this was introduced I think in Turbo Pascal 5 or so, but people seem to say that it wasn't until version 7 that Borland had ironed out most of the bugs. I don't see myself using any OOP features that much (if at all), for the same reasons I now stick to C. I just prefer the procedural approach.

The limitation of your code needing to run in DOS real-mode is unfortunate to me in 2018, but if anything this just enforces developing for DOS the way it really was done for most of it's life... with 640k conventional memory, heh. Of note, Borland Pascal 7 (from what I gather, is the "professional" version of Turbo Pascal) apparently did include some ability to have your code run under DPMI and also added some 32-bit / 386 assembly support. However, I've read enough bad things about Borland's DPMI support in general that I'm not particularly interested in trying it out for myself.

For my current "retro" programming projects, I don't see myself using Turbo Pascal to tackle some of my more ambitious ideas (such as the Doom-style engine I still want to try doing), but for simple 2D graphics stuff I actually think this will be an interesting alternative to C.

The quicker edit-compile-run cycle is definitely handy. It makes prototyping or otherwise just quickly experimenting with ideas much easier. On my 486, it feels like instant compile times except maybe if I'm doing a full rebuild (which still completes in maybe two seconds). Contrast that to Watcom C where even for simple changes to a single source file, you're still waiting at least several seconds (if not longer). It makes a big difference over a day spent working on a project. I guess that is why many people who do retro programming today tend to use DOSbox or something else on their modern computers. I still refuse to go down this road though, preferring to stick with completely era-appropriate software and hardware!

Updated libDGL Code

April 28, 2018 code dos

A quick post just to point out that I updated the libDGL Github repository with the most up to date working code that I currently have.

Since I originally pushed libDGL code to Github last November, not much new functionality/features has been added. Kind of disappointing for me to think about actually, heh. That being said, over all that time, I do feel like I fixed up a bunch of bugs and generally improved the performance of what was there. However, looking at my to-do list that is left for libDGL, I still really have my work cut out for me:

  • Scaled/rotated blitting support
  • Blending
  • "Mode 7" like support
  • Custom font loading (BIOS-like format?)
  • Joystick / Gravis GamePad support
  • Input device (keyboard/mouse/joystick) events
  • PC speaker sounds
  • Sound Blaster compatible sound/music
  • Gravis Ultrasound compatible sound/music
  • Sine/cosine lookup table optimizations
  • BMP, LBM, GIF image loading (and saving?)
  • Simple immediate mode GUI

This list is definitely not in any particular order. I want to start building a simple 2D map editor tool (since the old QBasic one I have sitting here is missing source code, so I cannot even just extend it as a quick alternative), so the last item about a "simple immediate mode GUI" is probably going to be my next task.

Following that, I kind of what to do something with audio. I've been focusing a lot on graphics lately and feel like a change would be nice. More specifically, I think starting with some MIDI playback might be fun. I just recently picked up a Roland Sound Canvas SC-88VL (through which, MIDI songs sound absolutely exquisite) and this is most probably influencing that decision, heh. However, I think I'd likely want to start with writing MIDI playback code for a Yamaha OPL as that was far more commonplace, but supporting general MIDI devices also sounds like a nice second step.

Using Watcom's Register-based Calling Convention With TASM

April 16, 2018 code dos

I suppose I'm writing this post for my own benefit primarily. I'll likely forget many of these details in a month, and then go and try to write a bunch more assembly and run into problems. So I'll try to proactively solve that future problem for myself. Everything here is better documented in the compiler documentation. However, it is scattered around a bit and of course isn't written with specific examples for using TASM.

One of the performance benefits that Watcom brought with it that was a pretty big deal at the time was that it's default calling convention used registers for up to the first 4 arguments to called functions. Past that, and the stack would be used as per standard C calling conventions.

As mentioned this calling convention is the default, but it can be globally changed via the CPU instruction code generation compiler switch. For example, /3 and /3r both select 386 instructions with register-based calling convention, while /3s selects 386 instructions with stack-based calling convention.

Borland Turbo Assembler (TASM) does not natively support this register-based calling convention among it's varied support for programming-language specific calling conventions. However it does let you use it's "NOLANGUAGE" option (which is the default if no language is specified) and then you can handle all the details yourself.

ideal

p386
model flat
codeseg

locals

public add_numbers_

; int add_numbers(int a, int b)
; inputs:
;   eax = a
;   edx = b
; return:
;   eax
proc add_numbers_ near
    push ebp
    mov ebp, esp

    add eax, edx

    pop ebp
    ret
    endp

end

This is pretty normal looking TASM. Complete with normal looking assembly prologue and epilogue code. Note that we are intentionally not specifying a language modifier.

So, first off, add_numbers_ has a trailing underscore to match what Watcom expects by default. If you don't like this for whatever reason, you can change the name here to your liking, but the use of a #pragma in your C code is necessary to inform Watcom about the different naming convention for this function.

Second, via the magic of the register-based calling convention, Watcom will have our two number arguments all ready for us in eax and edx. Our return value is assumed to be in eax, and that is correct in our case so we're all good.

The great thing is, we don't actually need to do anything fancy to call this function from our C code.

// prototype
int add_numbers(int a, int b);

// usage
int result;
result = add_numbers(10, 20);

But that was the simple case.

This register-based calling convention actually places the burden on the called function to clean things up before returning. This includes preserving some register values as well. According to the documentation: "All used 80x86 registers must be saved on entry and restored on exit except those used to pass arguments and return values." So, in our add_numbers_ function if we had wanted to use ecx, we would need to push and pop it during the prologue and epilogue code. But we didn't need to do so for eax and edx because those were used to pass arguments and return a value.

As mentioned previously, the stack gets used for arguments once all the registers have been used for arguments (by default, eax, edx, ebx, ecx in that order). In this case, the called function is responsible for popping them off the stack when it returns. So, if there were two int arguments that were passed on the stack, we would need to do a ret 8 to return.

; For this function, using the default register calling convention, the first 4 arguments
; will be passed in registers eax, edx, ebx and ecx. The last two will be passed on the stack.

; void direct_blit_4(int width4,
;                    int lines,
;                    byte *dest,
;                    byte *src,
;                    int dest_y_inc,
;                    int src_y_inc);
proc direct_blit_4_ near
arg @@dest_y_inc:dword, @@src_y_inc:dword
    push ebp
    mov ebp, esp  ; don't try to be clever and move this elsewhere!
    push edi      ; likewise, don't try to group the push's all together!
    push esi

    ; code here (that also modifies edi and esi, thus the additional pushs/pops)

    pop esi
    pop edi
    pop ebp
    ret 8
    endp

Is this all too cumbersome to worry about? Well, I don't really think it's a big deal, but there is a way we can remove ourselves from this burden.

Let's say we didn't want to have to worry about preserving any of eax, ebx, ecx, edx, edi, or esi regardless of how many arguments our function has and what (if any) return value it uses. Also, maybe we don't want to have to worry about popping arguments off the stack ourselves when our assembly functions return.

// define our "asmcall" calling convention
#pragma aux asmcall parm caller \
                    modify [eax ebx ecx edx edi esi];

#pragma aux (asmcall) add_numbers;
int add_numbers(int a, int b);       // no change to the function prototype is necessary

What if we actually wanted to use the normal C stack-based calling convention for our assembly functions and ignore this register argument nonsense? Maybe you're using an existing library and it was written for other compilers that don't use this register-based calling convention.

#pragma aux asmstackcall parm caller [] \
                         modify [eax ebx ecx edx edi esi];

Watcom also pre-defines the cdecl symbol for this same purpose, which you can and probably should use instead of defining your own.

The empty brackets [] denotes an empty register set to be used for parameter passing. That is, we are saying not to use any registers, so the stack is used instead for all of them. With that in mind, we could expand the set of default registers used for parameter passing:

#pragma aux asmcallmorereg parm caller [eax edx ebx ecx edi esi] \
                           modify [eax ebx ecx edx edi esi];

In this case the modify list is redundant and need not be specified.

Of course, saying that your function will use/modify more registers means that the compiler has to work around it before and after calls to your assembly function which may result in less optimal code being generated. There's always a trade off!

None of the above #pragmas remove the need for the standard prologue and epilogue code that you've seen a thousand times before:

push ebp
mov ebp, esp
; ...
pop ebp

The only exception is if your assembly function isn't using the stack at all.

There are many details I've left out. For example, passing double values will mean two registers will get used for one argument because doubles are 8 bytes. But if you only have one register left (maybe you passed 3 ints first), then the double value will get passed on the stack instead. Additionally there are more details to know when passing/returning structs. But I'm not doing any of this right now, so I've not really looked into it beyond a passing glance.