Exploring Watcom C/C++

A couple weeks ago I picked up a complete boxed copy of Watcom 10.0a. I was also able to find an almost complete set of the printed documentation/manuals for Watcom 9.5 from a somewhat-local seller.

I do love having physical copies of books, especially reference material like this. Most people will probably think I'm weird, but I find this more convenient to refer to in many (but not all) cases. Being able to physically flip through pages at will and back and forth between several different pages as needed just feels a ton more convenient to me then doing the same with a PDF. And I've always preferred reading text like this on real paper. The obvious downsides versus an electronic copy is the lack of ability to do a Ctrl-F to find something specific and also not being able to copy+paste something from it.

So what's so special about Watcom anyway? Well, for me personally, it was the first C compiler I ever used. Not this version though. In 1996 I remember my dad bought me a book, "C DiskTutor" by L. John Ribar (published in 1992) that came with a disk containing sample code and a stripped down copy of the Watcom C 8.5 compiler. The disk actually had a few bad spots on it and I remember he called the publisher and they sent out another disk. I didn't realize it at the time, but the version of the compiler on the disk lacked the ability to compile code with any memory model other then the small model, so it wasn't suitable for really any "real world" use. Certainly good enough for introductory exercises though, which was obviously the intended purpose.

I didn't spend a whole lot of time using it though. I remember getting frustrated with fixing my own bugs as there was no included debugger with this stripped down version. At least I don't remember there being one. But also I remember trying to make sense of the poor compiler error messages was tough for me being completely new to C at the time. Finally, I also remember really hating how I had to quit out of EDIT.COM to compile and run my code. The disk included some simple text editor which seemed to be written specifically for this book. It did have the ability to run the compiler from within the editor, but I vaguely remember arriving at the conclusion that this editor was not very good and a bit buggy. EDIT.COM ended up being a less-frustrating option for writing code. Keep in mind my only other programming experience at this time was a little over a year with QBasic which had a much faster and easier edit/compile/run cycle. I didn't understand why anyone would want to use something else when it was so much easier! Of course years later I learnt that many people at that time used better editors then EDIT.COM and actually really liked writing code in their editor of choice and using external build scripts / Makefiles. Heh.

Beyond my nostalgia though, Watcom was used by all sorts of game developers who cared about code efficiency in the early-to-mid 90's, such as id Software with Doom and 3D Realms with Duke Nukem 3D amongst many others. Plus it shipped with a royalty-free version of the well known DOS extender DOS/4GW. However, while it was better at code generation then the competition (Borland, Microsoft, etc), it lacked in the end-user experience. Borland and Microsoft had better tooling in the form of IDE's (Watcom didn't ship an IDE with their compiler until 10.0) and easier to use command line tools.

Eventually other compilers caught up and Watcom converted to an open-source project when Sybase (who owned Watcom) decided to end-of-life the compiler.

So far I've been using DJGPP (which is just a DOS port of GCC) for all my DOS coding projects. I'm using an old version that corresponds roughly with what was current when I first discovered it for no other reason then that it feels "true" to the whole spirit of these DOS coding projects.

DJGPP is pretty awesome honestly. Since it's just an older version of GCC for DOS, it feels very familiar. Lots of the same tools are present, including GDB and MAKE. And the code generation is really quite excellent. I think my only gripes with DJGPP are that it uses the AT&T syntax for inline assembly and that I kind of have a tiny bit of a love/hate relationship with the CWSDPMI DOS extender and DPMI server it uses. It's nothing major really, just minor gripes relating around how it seems more "complex" to use on the surface as compared to something like DOS/4G. But I realize that there are good reasons for it. And personally I believe that CWSDPMI is probably a bit more robust then DOS/4GW, but I could be wrong. Feels that way sometimes though.

RHIDE is an IDE for DJGPP that seems to be largely inspired by Borland's DOS IDEs. It's also pretty great and is very quick to get up and running. You don't even need to do any fiddling about with GCC or MAKE directly when using RHIDE as it will handle that all for you. Awesome. Though it's not without its quirks and limitations. The editor has some weird (to me) ideas about indentation that I've only been able to "resolve" by using a very specific combination of options that involves mixing tabs and spaces in a way that makes me sad.

Additionally, while the GDB integration is good, it definitely feels like it's missing some extra polish. There's no automatic locals watching (have to explicitly add anything you want to watch the value of). The watch viewer itself is quite simplistic and watching the values of structs is ... a poor experience. Well, not until you start wanting to watch more complex values anyway. The whole watch window really needed to be implemented as an expanding/collapsing node tree type of viewer, but it's not sadly. And there's no memory viewer. The disassembler viewer really lacks a source code + assembly code combined mode. It's not the end of the world, as when you're stepping through code you can see both and the current line is highlighted in both places. But still... it makes browsing the assembly view somewhat more annoying than it needed to be. Bleh. Oh, and there really isn't any better GDB frontend for DJGPP/DOS that I'm aware of. This is pretty much the best option. And to be fair, for the time, it wasn't bad at all. It's just that my nostalgia only goes so far.

So anyway, back to Watcom. The version I picked up (10.0a) is 4 years older then the version of DJGPP I've been using so far. So I didn't have any expectations about getting a "better" compiler or anything like that. But I was pretty excited to try it out because it really feels much more "true" to the spirit of using a 486 to do DOS programming as Watcom was the compiler that the majority of my favourite DOS games were written with. A silly reason, for sure, heh.

I'd heard the "new" IDE that Watcom introduced with 10.0 was kind of not good, and certainly that was my reaction to trying it out as well. It requires Windows 3.x and it does work. But it's pretty weird. The "IDE" is actually comprised of a series of separate executables that are all invoked from the main "IDE" (what is actually called the "IDE" by Watcom). It's more of a "project manager and GUI build tool" then anything else. You can see all the source files in your project and configure various compile and link options. Clicking on a source file causes a separate text editor application (and hence, a separate window) to open. Same goes with things like Windows resource files, etc. All separate tools. It's pretty weird. No other IDEs even at that time were like that. Certainly not any of the well known ones anyway. Kind of makes me curious what their reasons were for going with this design. Anyway, it's certainly not something I want to use. Additionally, I found that debugging graphical DOS programs launched from within Windows was problematic and resulted in crashes sometimes while stepping through code. So the Watcom IDE is out.

I had to admit at this point that I was really quite unfamiliar with what DOS text editors were out there. Turns out there was quite a lot. Many even had fancy features like syntax highlighting. Some also had fancy macro languages for additional configurability. Nice. Should be easy to find something suitable.

Heh.

It would take too long to write fully about this experience, but suffice it to say that while I didn't try everything on that list, I did try quite a lot. I realized that I am quite picky about the text editor I use. In weird ways anyway, probably mostly born out of habit and fear of change. Also anything based on VIM or Emacs I disregard because that's just how I roll.

I basically wanted the following features at an absolute minimum:

  • Syntax highlighting.
  • Multiple files open at once and able to easily switch between them.
  • External tools launching (to launch batch scripts / makefiles to run/compile/debug).
  • Ability to run in 80x50 text mode.

Not too much to ask for I think. Multiple editors had support for all of these in fact. The first one I decided to look at was SETEDIT because it's what RHIDE was made from (or is that the other way around?). I quickly discovered that launching DOS/4G programs from within CWSDPMI programs can be problematic. The DOS version of SETEDIT is built with DJGPP which means it uses the CWSDPMI DOS extender. Programs that I was going to be building with Watcom would make use of DOS/4GW. Additionally some of the Watcom tools, such as the DOS debugger WD, also use DOS/4GW. I was unable to launch the Watcom WD debugger from within SETEDIT without causing crashes. While some test programs I built which also used DOS/4GW could be launched fine, I didn't like the idea that down the road there might be problems. Ugh.

As it turns out, a lot of the better text editors were also built with DJGPP. All of these that I tried which supported launching external tools/programs also exhibited the same crashing problems when launching some DOS/4GW programs like WD. Ergh.

I do want to specifically say that I really liked FTE. Just a great, simple, minimalistic, easy to use editor which for some reason really resonated with me. However, it is built with DJGPP so it was sadly a no-go. That being said, I have a "conversion" project on my backlog list now, heh. If I decide to use Watcom over the long term, I intend to fully investigate whether it's possible to convert FTE to compile with Watcom so as to hopefully remove the crashing issues with DOS/4G programs.

Another editor I wanted to point out is something called "Power View IDE" which is actually a full DOS IDE for Watcom C/C++ that some guy wrote! I was ecstatic when I found this linked on some random forum post, seemingly as an afterthought by the poster. I don't think there is any existing project website for it, but it can be downloaded here. I couldn't believe that there was so little information out there about this and basically no one ever mentioned it anywhere.

This editor I think could have been exactly what I was looking for. Except it wasn't. It had some show-stopping bugs which made me oh so very sad. For one, when scrolling through text using the arrow keys on the keyboard (as I sometimes do), the editor would periodically insert random numbers ... !? What the heck? Maybe a bug in the keyboard handler? I took a look at the code and nothing stood out to me, so not sure what to make of that one. But yeah, an editor that randomly inserts random characters into your code is instantly disregarded. Another issue was that it didn't seem to correctly handle restoring the video mode when returning from executed programs. This became problematic when running the editor in 80x50 text modes which was a requirement of mine. Oh well. I guess the buggy nature of it is why no one ever really seemed to talk about it from what I could see.

To make this discussion of text editors not go on for many more paragraphs I'll cut to the chase and say that I eventually settled on Aurora. It was actually my last resort option, heh. While it has quite a lot of weird quirks that were initially off-putting for me, it is an amazingly customizable editor due to it's weird macro language "AML." Everything in the editor itself is largely implemented in AML, from how text editor behaviours work, to the menus, to the keyboard shortcuts, to file open/close dialogs, etc, etc. The DOS executable is basically just a runtime environment for AML and the Aurora text editor just comes with a pre-built set of compiled scripts that give you a text editor. Neat. It's reasonably well documented and the macro language has support for quite a lot. It appears to be a 16-bit real-mode DOS executable so no problems executing DOS/4GW programs from within the editor.

However, it had no real built in support for some nice "quality of life" things such as capturing build errors and displaying them nicely in the editor. "Well, with all the things AML supports, how hard can it be to implement this myself in exactly the way I want it?" This line of thinking kind of spiraled out of control quickly. Suffice it to say, there were way more quirks with AML then I had expected and the debugging facilities within Aurora were barebones at best. I think the hardest part was actually managing the lifecycle of a plugin that is meant to be called again and again from a menu item. It wasn't straightforward and was complicated by the fact that the description of how AML's "objects" work made me think more of traditional OOP style programming ... and it kind of isn't like that in a lot of ways. The extensive documentation unfortunately doesn't really tell you anything about actually building plugins for the editor specifically. Nothing in the way of concrete examples with explanations for how to do common things or why it's done this way, etc. Instead, it tends to just focus on the language basics and some API stuff... nothing to tie it all together. Oh well. I should be happy that it was as well documented as it was really. After suffering through some bizarre and hard to debug editor crashes from poor lifecycle management, I got something working!

At this point, I had put off "fixing" all the subjectively weird behaviours and keybindings that Aurora has out of the box (even when selecting a keybinding scheme during installation that was supposedly based on an editor I was mostly familiar with). Finally got that all mostly sorted out and at this point was thinking to myself "wow, this REALLY all had better have been worth it!" Meaning of course, that I really hoped I wouldn't grow to hate Watcom, heh.

The "WMAKE" menu I added contains options for Build, Rebuild, Clean, Run, Debug and Show Errors. Build errors will show up automatically after any action has finished, but it's a quick key-bound menu item to easily bring back up the error window if it gets hidden. These WMAKE actions basically require a makefile that contains targets for each menu item. Which is totally fine, as that was basically how I envisioned the makefile structure I was going to use anyway.

target_config = debug

target_name = dgl_test

object_files = &  
    blit.obj &
    clipping.obj &
    dgl.obj &
    draw.obj &
    gfx.obj &
    internal.obj &
    keyboard.obj &
    mathext.obj &
    mouse.obj &
    pcx.obj &
    util.obj &
    test.obj

cc_flags_debug = /d2 /zp4 /5r /fp3  
cc_flags_release = /d1+ /zp4 /5r /fp3 /oneatx  
cc_flags = /mf $(cc_flags_$(target_config))

link_flags_debug = debug all  
link_flags_release = debug all  
link_flags = $(link_flags_$(target_config))

.c.obj: .AUTODEPEND
    wcc386 $[. /zq $(cc_flags)

$(target_name).lnk: $(object_files)
    %create $^@
    %append $^@ NAME $(target_name).exe
    %append $^@ SYSTEM DOS4G
    %append $^@ OPTION QUIET
    @for %i in ($(object_files)) do %append $^@ FILE %i

$(target_name).exe: $(object_files) $(target_name).lnk
    wlink $(link_flags) @$(target_name).lnk

clean : .SYMBOLIC  
    del *.obj
    del *.err
    del $(target_name).exe
    del $(target_name).lnk

.NOCHECK
build : $(target_name).exe

.NOCHECK
run : $(target_name).exe  
    $(target_name).exe

.NOCHECK
debug : $(target_name).exe  
    wd /swap /trap=rsi $(target_name).exe

Basically relying on WMAKE to launch what I need once builds complete.

If that one other weirdo like me who wants to use Watcom C/C++ with the Aurora Text Editor comes by this site and reads this, here's my WMAKE.AML plugin. The two main things that are currently missing are "click on error to goto sourcefile/line" support and some kind of better window management (like, e.g. automatically resizing existing editor windows to have the errors window always visible on screen). The former will likely be added soon as it should be simple to add.

So anyway ... surely I would not have to all of this effort without having some idea of whether it was going to be worth it at the end right? Of course. Heh.

I'm quite a big fan of the Watcom debugger WD (previously called WVIDEO... not sure where the "video" part of that name comes from honestly).

It's kind of got a bit of a Visual Studio C++ debugger thing going on that I like (and this was even from before Visual Studio C++'s debugger got good I think). I think the one thing I dislike about it so far is that abruptly terminating a running program can leave your system in a weird state. For example, it doesn't seem to always restore overridden interrupt handlers. So not letting your running code run to the point where it removes it's custom keyboard interrupt handler can leave you unable to do anything once the debugger exits and returns to your text editor. Oops. DJGPP and GDB (and maybe CWSDPMI?) seemed to do a much better job of cleaning things up in these cases so your system didn't crash very often (though it wasn't perfect).

As for Watcom in general, I like that DOS/4GW seems to get out of your way a little bit more than CWSDPMI does. For example, the first 1MB of memory is directly accessible out of the box, so I can write into 0xA0000 immediately, or read directly from some memory in the BIOS to e.g. read the RTC tick count, or the BIOS fonts, etc. Just as DOS should be! Additionally, I can just call _dos_setvect and _dos_getvect directly as I would in a 16-bit realmode program and it will basically "just work" when I want to implement my own interrupt handler. Of course, it's not quite that simple in all cases, like when you want to use some bit of shared DOS memory and communicate with some process or piece of hardware that is running in 16-bit mode. But for the simple cases, it is simple. Don't need to have a bunch of _go32_dpmi_blahblah calls around. This will come back to bite me in the ass though, as DJGPP provides a relatively nice API for interfacing with CWSDPMI when you (inevitably) need to for the more complicated scenarios. DOS/4GW doesn't provide you with as nice an API... you just call the standard DPMI server interrupt. However, this is mostly documented in my stack of Watcom manuals, so that's a nice plus! Even some examples such as how to implement bimodal interrupt handlers.

On the topic of code generation though, the version of GCC from DJGPP that I was using is definitely better (as expected, it is 4 years newer).

We can take a look at the following very simplistic example which shows how something simple like function inlining is handled between these two compilers. In this example, I was using GCC flags -O2 -m486 -ffast-math and WCC386 flags /zp4 /5r /fp3 /oneatx both of which are recommended flags for optimum performance (could do -O3 with GCC of course though).

This example is something I noticed while running some simplistic benchmarks to see how much performance I would lose with Watcom. This snippet isn't even what I was benchmarking... it's just some setup code to generate a random 16x16 sprite for the actual benchmark that appeared later in the code. However I noticed it wasn't inlining a call the way I was expecting when looking at the assembly output.

// in a header file
static int surface_offset(const SURFACE *surface, int x, int y) {  
    return surface->width * y + x;
}

static byte* surface_pointer(const SURFACE *surface, int x, int y) {  
    return surface->pixels + surface_offset(surface, x, y);
}

static void surface_pset_f(SURFACE *surface, int x, int y, int color) {  
    *(surface_pointer(surface, x, y)) = (byte)color;
}

// elsewhere, in main()

bmp = surface_create(16, 16);  
for (x = 0; x < 16; ++x) {  
    for (y = 0; y < 16; ++y) {
        surface_pset_f(bmp, x, y, rand()%255);
    }
}

As a quick aside, before anyone looks at the above code and says anything like "Wow, that would be slow anyway! You really don't want to be using anything like an explicit pset type of function call anywhere! Look at that slow multiplication for god's sake!" I know. For any code that's running in an inner loop in a performance-sensitive scenario, I would always use, e.g. the above surface_pointer function to get a direct pointer to the memory before the loop and then use that pointer in the inner loop itself. And if it was really important to save every last CPU cycle, I would probably write out the surface_offset calculation directly without the multiplication anyway. These are all just convenience functions to be used as appropriate. The above for loop code is just an example!

GCC inlines the call to surface_pset_f as I would expect:

  21:test.c        ****     bmp = surface_create(16, 16);
 185 007e 6A10             pushl $16
 186 0080 6A10             pushl $16
 187 0082 E879FFFF         call _surface_create
 187      FF
 188 0087 8945E8           movl %eax,-24(%ebp)
  22:test.c        ****     for (x = 0; x < 16; ++x) {
 190 008a 31F6             xorl %esi,%esi
 191 008c 83C410           addl $16,%esp
 192 008f 90               .align 2,0x90
 193                  L238:
  23:test.c        ****         for (y = 0; y < 16; ++y) {
 195 0090 31DB             xorl %ebx,%ebx
 196 0092 89F6             .align 2,0x90
 197                  L242:
  24:test.c        ****             surface_pset_f(bmp, x, y, rand()%255);
 218 0094 E867FFFF         call _rand
 218      FF
 219 0099 89C2             movl %eax,%edx
 220 009b BFFF0000         movl $255,%edi
 220      00
 221 00a0 99               cltd
 222 00a1 F7FF             idivl %edi
 223 00a3 8B45E8           movl -24(%ebp),%eax
 224 00a6 8B38             movl (%eax),%edi
 225 00a8 0FAFFB           imull %ebx,%edi
 226 00ab 01F7             addl %esi,%edi
 227 00ad 037808           addl 8(%eax),%edi
 228 00b0 8817             movb %dl,(%edi)
 230 00b2 43               incl %ebx
 231 00b3 83FB0F           cmpl $15,%ebx
 232 00b6 7EDC             jle L242
 234 00b8 46               incl %esi
 235 00b9 83FE0F           cmpl $15,%esi
 236 00bc 7ED2             jle L238
  25:test.c        ****         }
  26:test.c        ****     }

However, I noticed wcc386 did not. Not only that, no amount of tweaking of compiler settings and/or #pragmas would get it to. It only partly inlines the call to surface_pset_f:

    bmp = surface_create(16, 16);
 142a  ba 10 00 00 00                    mov     EDX,00000010H
 142f  89 44 24 04                       mov     +4H[ESP],EAX
 1433  89 d0                             mov     EAX,EDX

    for (x = 0; x < 16; ++x) {
 1435  31 f6                             xor     ESI,ESI
 1437  e8 00 00 00 00                    call    surface_create_
 143c  89 c7                             mov     EDI,EAX

        for (y = 0; y < 16; ++y) {
 143e  31 c9             L27             xor     ECX,ECX

            surface_pset_f(bmp, x, y, rand()%255);
 1440  e8 00 00 00 00    L28             call    rand_
 1445  89 c2                             mov     EDX,EAX
 1447  bb ff 00 00 00                    mov     EBX,000000ffH
 144c  c1 fa 1f                          sar     EDX,1fH
 144f  f7 fb                             idiv    EBX
 1451  89 14 24                          mov     [ESP],EDX
 1454  89 cb                             mov     EBX,ECX
 1456  89 f8                             mov     EAX,EDI
 1458  89 f2                             mov     EDX,ESI
 145a  e8 00 00 00 00                    call    surface_pointer_
 145f  8a 14 24                          mov     DL,[ESP]

        }
 1462  41                                inc     ECX
 1463  88 10                             mov     [EAX],DL
 1465  83 f9 10                          cmp     ECX,00000010H
 1468  7c d6                             jl      L28

    }

 146a  46                                inc     ESI
 146b  83 fe 10                          cmp     ESI,00000010H
 146e  7c ce                             jl      L27

Elsewhere in that same module, surface_pointer_ can be found:

0070                    surface_pointer_:  
 0070  0f af 18                          imul    EBX,[EAX]
 0073  8b 40 08                          mov     EAX,+8H[EAX]
 0076  01 da                             add     EDX,EBX
 0078  01 d0                             add     EAX,EDX
 007a  c3                                ret     
 007b  90                                nop     

So, Watcom partially inlined the call to surface_pset_f but left a call to surface_pointer as a fairly lightweight function call. Note how Watcom uses registers to pass parameters to functions pretty much everywhere it can by default, thusly the surface_pointer_ function has none of the typical stack management boilerplate we might expect. From what I've read, this method of passing function parameters is one of the reasons it kept ahead of the pack regarding performance compared to other compilers at the time.

Whilst playing with compiler flags and other things, I decided to see how wpp386, the Watcom C++ compiler, would handle this:

    bmp = surface_create(16, 16);
 0063  ba 10 00 00 00                    mov     EDX,00000010H
 0068  89 44 24 08                       mov     +8H[ESP],EAX
 006c  89 d0                             mov     EAX,EDX

    for (x = 0; x < 16; ++x) {
 006e  31 f6                             xor     ESI,ESI
 0070  e8 00 00 00 00                    call    __16hkttSURFACE near * near surface_create( int, int )
 0075  89 c1                             mov     ECX,EAX
 0077  eb 06                             jmp     L3
 0079  46                L2              inc     ESI
 007a  83 fe 10                          cmp     ESI,00000010H
 007d  7d 2a                             jge     L5

        for (y = 0; y < 16; ++y) {
 007f  31 db             L3              xor     EBX,EBX

            surface_pset_f(bmp, x, y, rand()%255);
 0081  e8 00 00 00 00    L4              call    rand_
 0086  89 c2                             mov     EDX,EAX
 0088  c1 fa 1f                          sar     EDX,1fH
 008b  f7 fd                             idiv    EBP
 008d  89 14 24                          mov     [ESP],EDX
 0090  8b 11                             mov     EDX,[ECX]
 0092  0f af d3                          imul    EDX,EBX
 0095  8b 41 08                          mov     EAX,+8H[ECX]
 0098  01 f2                             add     EDX,ESI
 009a  01 c2                             add     EDX,EAX
 009c  8a 04 24                          mov     AL,[ESP]

        }
    }

 009f  43                                inc     EBX
 00a0  88 02                             mov     [EDX],AL
 00a2  83 fb 10                          cmp     EBX,00000010H
 00a5  7d d2                             jge     L2
 00a7  eb d8                             jmp     L4

Interesting! wpp386 fully inlines surface_pset_f. The exact same compiler flags were used in both cases.

After discovering this difference, I decided to compare wpp386 vs wcc386 with some other pieces of code and saw that there are definitely some other code efficiency differences. I've not investigated fully yet, but it does seem like wpp386 generates slower code in some cases too. I'll probably do a follow up post once I've dug into it a little bit more. One of the things I want to play with is re-working the code for surface_pset_f, surface_pointer and surface_offset to see if it's just a problem caused by the code optimizer "giving up" at some point because of how the code is written. It definitely was a thing that you'd have to at times help the compiler out by simplifying or otherwise breaking up complex expressions into separate chunks so it could do a better job.

Once nice thing about using wpp386 for even 100% C code is that it at least gives you more warnings and the support for toggling off warnings that you don't care about (like "W202" with wcc386 ... ugh) actually works. Turning up the warning level with wcc386 is kind of useless for this reason. I'm assuming this is just a bug and was probably fixed with 10.5 or 11.0.

Inline assembly is similar in some ways to GCC, but no need to deal with the AT&T syntax which I dislike. The way you do it with Watcom 10.0 is unfortunately more complicated then using asm as you might with other C compilers of the time (and modern compilers too), but it has it's benefits too:

void REP_MOVSD(void *src, void *dest, int num_dwords);  
#pragma aux REP_MOVSD =     \
    "cld"                   \
    "rep movsd"             \
    parm [esi] [edi] [ecx]  \
    modify [esi edi ecx];

Then you could simply call REP_MOVSD in your C code in the exact same way you would call any other C function. Inline assembly written in this manner would be inserted into your code at compile time in the same way as an C function that is being inlined by the compiler would be.

Compare to how you'd do it with DJGPP/GCC:

#define REP_MOVSL(src, dest, num_dwords)            \
    __asm__ __volatile__ (                          \
        "cld\n\t"                                   \
        "rep\n\t"                                   \
        "movsl"                                     \
        : : "S" (src), "D" (dest), "c" (num_dwords) \
        : "%ecx", "%esi", "%edi" )

Works the same way when calling from your C code (REP_MOVSL can be called as any other function, and is inlined, etc.). Also the GCC __asm__ is more flexible like what you saw with other compilers and the asm keyword, so you could just use it wherever in your code with ease (so you don't have to use it in a macro as I'm doing here).

If I'm being honest, both methods are super ugly to me. I will never be able to commit either form to memory and so, every time I need to write inline assembly I'll need to be referring to some documentation to copy it out. Watcom 11.0 apparently added support for the asm keyword, but apparently even with that, Watcom recommended the #pragma method because of its ability to indicate to the compiler what registers are used to pass values in and what registers were modified by the inline assembly code, something that the compiler couldn't do with the asm keyword I guess.

libDGL

Been a while since I wrote about this. When it comes to my own personal coding projects, rest assured that I take it very slowly. Heh.

I released the code that I've been working on for the past month. It's a DJGPP library called libDGL. DGL stands for "DOS Gamedev Library." Yes, I am incredibly unoriginal when it comes to naming things. This is a library aimed at "retro game development" with MS-DOS, targeting VGA Mode 13h using C.

As is mentioned on the Github project page linked above (and as I've mentioned previously here as well), I am using an older version of DJGPP from the late 90's. More specifically, I am using:

15 Jan 1998   bnu281b.zip  
24 Oct 1997   csdpmi4b.zip  
31 Oct 1996   djdev201.zip  
18 Jan 1997   faq210b.zip  
 6 Jun 1998   gcc281b.zip
18 Oct 1996   gdb416b.zip  
 6 Jun 1998   gpp281b.zip
 1 Mar 1998   mak3761b.zip
30 Sep 1997   rhide14b.zip  

I make no guarantees that this code will work with different versions. I'll probably test it at some point, but for now I am more interested in fixing bugs and adding more features. My "todo" list for this library is quite long still. Even so, I do feel like I've got a fair bit accomplished so far.

To help me test out a bunch of this, I wrote a very simple Asteroids game over the past two days. Asteroids is a very simple game, and I feel like two days was a long time to take to write it, but in the process I uncovered and fixed a number of bugs in libDGL, so I guess I should not feel like I was too slow. Finding and fixing bugs was the WHOLE point of writing it after all.

The code is available here.

As you can imagine based on the above screenshot and the game being Asteroids, it's nothing particularly special, heh. In fact, this has not been tweaked to provide any real level of difficulty to the player at all. I was more interested in testing out libDGL then in balancing a game and providing a full layer of polish. As well, I am less than happy with how the code that handles the different game states turned out. It's fairly sloppy honestly, heh.

This game doesn't make use of any sprite blitting. Instead, it uses line drawing and 2D vector transformations for the graphics. This was useful to test out and verify the math functions I had written, and is the main reason I picked Asteroids.

Not really much to say about it honestly. The next test game I'd like to do is probably some kind of simple vertical 'shmup type of game using sprites for graphics. Probably something along the lines of what I was originally going to do on the Amiga 500 some months ago.

So, where am I going with all of this anyway? Well, I don't have any specific plan worked out, but in the back of my head I've got some grandiose ideas about writing some 2D dungeon crawler type game (something I wanted to do as a kid back in the 90's but never finished... actually, that might be a fun post to write in the future, revisiting some of that code from back then which I have sitting here now). As well, I'd like to eventually work my way up to some 3D raycasting games, with a final goal being something Doom-like but with some RPG elements thrown in (and not gritty/dark like Doom is). But this is all quite a long ways off, and first thing's first... gotta work on the foundation.

DOS coding

Last month I picked up a copy of Fabien Sanglard's Game Engine Black Book: Wolfenstein 3D. In fact, I was eagerly waiting for the day I could order a copy. When it arrived I was totally engrossed in it the whole way through. It's an amazing book, well written and well worth the read for anyone interested in those kinds of topics. Fabien really did a great job with it and I'm eagerly awaiting the next book in the series which will cover the DOOM engine.

It also got me thinking about some DOS code I had started to write a couple months prior and then set aside temporarily. I had begun writing a simple VGA Mode 13h library I had called "DGL" for "DOS Game Library" because naming is not a strong suit of mine. So, I'm going to pick it up again and hopefully continue writing about it here as I work on it and then soon after, some little game demos written with it also.

Of course, there's absolutely no good reason to re-invent the wheel from scratch like this. Libraries such as Allegro exist and any of the 3.x or 4.x versions for DOS would perfectly meet all my requirements and is probably far better implemented than anything I'd cook up myself. But that would be less fun.

Anyway, what I want out of this library is:

  • Support for VGA mode 13h (320x200x256)
  • Primitive drawing (pixels, lines, boxes, circles, polygons, etc)
  • Bitmap/Sprite drawing (aka. "bitblit"-ing)
  • Palette manipulation (loading, rotating, fading)
  • BMP, PCX, IFF image support (period-correct file formats)
  • Font rendering (using BIOS font format). Also add non-fixed width support?
  • Keyboard, Mouse, Joystick input device support
  • PC Speaker sound (maybe even rip off QBasic's PLAY command?)
  • Sound Blaster (and compatible) support (MIDI music, FM synth and digitized audio sound effects)
  • Math function suite (vectors, matrix, etc)

The only thing in this list that I think is (currently) outside my skillset (but totally possible to learn, of course) is the Sound Blaster stuff... simply because I've actually never written an audio engine of any sort, and certainly not ever written code directly for Sound Blaster hardware.

In fact, the only time I've added audio to game projects of mine was via DirectX. Not sure how DirectX is nowadays, but I remember at the time I was using Visual Basic 6 and the libraries for it had play/pause/stop functions for both sound effects and background music out of the box, so it was extremely simple to hook into the projects I worked on. Most game projects of mine tend to get dropped before I get to the point where I need to add audio, heh. And in more recent years, it was always my intent to use something like SDL_Mixer to take care of the lower-level details anyway, but I just never got around to it.

So, needless to say, when it comes to audio for this project I have a bit of fundamentals learning work to do.

Also on the topic of audio, I would like to look into adding support for Gravis Ultrasound cards. These cards are interesting as I think they were the first sound card for PCs to have full hardware audio mixing support. As I understand it, Sound Blasters lacked hardware mixing capability, so game engines would do this in software. But this will be a nice optional extra for later if I feel like it.

Originally, I had planned to use Borland C++ for this and do it all in real-mode DOS code. That probably sounds like totally unnecessary pain to anyone who understands the differences that brings along with it. I even went and bought a copy of Borland C++ 4.0 off eBay:

However, after fiddling around with it for a while I remembered how easy it was to crash your system with real-mode C code and how the debuggers of the time were helpful, but not that helpful in tracking down these types of bugs. This was less clear in my memory prior to this because during this time period I was primarily using QBasic which obviously shielded you from these types of bugs much more.

So instead, I plan to continue using a period-correct version of DJGPP (meaning, a version from the late 90's). Better debugger support for dealing with these types of crash bugs. Still possible to crash your system in weird and wonderful ways of course, but more support than Borland for catching these bugs. Plus having a 32-bit flat memory model is obviously very nice.

I plan to keep the code for this library on Github somewhere when there is something to show for it.

Shmup Game First Steps

After playing with both AMOS and Blitz Basic, I initially had an overall poor impression of Blitz Basic mostly due to stability issues. This was disappointing to me because it seemed pretty clear that AMOS was a little bit slower, but at least it did seem more approachable.

Ultimately though, when I reflected on the various game ideas I had floating around in my head, I really figured I would end up quickly outgrowing AMOS from a performance perspective. So I decided I would look at Blitz Basic again, but try to see what I could do (if anything) to work around the stability issues I had run into earlier. I should note at this point that yes, I did have the runtime debugger enabled, so I don't think that was the cause of any of the crashes I was seeing. That was the first thing I checked when I was having issues. Some of the crashes weren't even occurring while a program was running anyway...

First, I decided that I would not use the "mousable labels" support. Blitz's editor "TED" pops up a side navigation pane showing your program's labels for quick navigation only if your program has any. A "mouseable label" is just an ordinary label that you would use with GOTO or GOSUB with the exception that it should start with a period, e.g. .renderPlayfield: instead of renderPlayfield:. I had noticed some random crashing that would occur when I was fiddling around with some of the sample programs that included these types of labels. Additionally, I had encountered a severe problem with the updated TED version included in the Blitz Support Suite ("SuperTED" I believe it was called? Or might have been the update before that?) where it would immediately crash when opening a program with any of these types of labels. Definitely seemed to me that the editor support for it was at least partially broken... at least for me, though I'm not sure what I could have done to cause these issues, heh.

Second, I had finished completely reading through the user guide (which is honestly a quick read, less than 100 pages) and had run across the recommendations and warnings for using "Blitz mode" (which you will undoubtedly want to use for performance reasons). Most importantly, any disk access or other concurrent OS processing that occurs while a program is running in Blitz mode (or as it first enters Blitz mode) can cause crashes! How about that. I suppose it's possible that this accounts for at least some of the instability I was encountering before. The user guide recommends doing something like VWait 120 just before a BLITZ call to ensure that any disk buffers have had time to be flushed (evidently the OS does so every two seconds?). It notes that there is no software method to check for whether stuff like disk activity has completed so this seems like the only way to protect against this kind of crash. Also it is important to note that none of the sample programs that use Blitz mode do this, thus furthering my belief that this was at least part of the problem I had run into when previously toying about with Blitz Basic.

Anyway... so far so good! I've only had one crash all weekend, and I believe that actually was related to my misuse of Blitz mode and/or of some dual playfield slice functionality. For most of the weekend, I was booted up into a Blitz Basic 1.7 floppy and not using the hard disk functionality provided by my ACA500plus. I wanted to remove as much other possible factors as I could and that seemed like a big one. Late this afternoon I resumed using the ACA500plus and CF card hard disk and have so far been running completely stable as well with that configuration. Fingers crossed that it stays that way.

Without having to worry about crashes, I was able to get a lot of experimentation and learning done and got some very preliminary work done on a basic vertical shmup type game. Nothing fancy, but I'm happy for a few hours work with lots of flipping around through manuals slowing things down (really want to get a physical copy of the Blitz Basic Reference Manual... I hate having to constantly look up commands in PDFs).

I'll try not to make posting big code dumps a habit on this blog, but here's what I've got so far:

; shmup test

#KEY_ESCAPE = $45
#KEY_UP = $4c
#KEY_DOWN = $4d
#KEY_LEFT = $4f
#KEY_RIGHT = $4e
#KEY_SPACE = $40

#SPEED = 2
#STATUSBAR_HEIGHT = 32
#PLAYFIELD_HEIGHT = 256 - #STATUSBAR_HEIGHT

#PAL_GAME = 0

#BMP_STATUSBAR = 0
#BMP_PLAYFIELD = 1

#SLICE_STATUSBAR = 0
#SLICE_PLAYFIELD = 1

#SHAPE_LRG_ENEMY = 0
#SHAPE_MED_ENEMY = 2
#SHAPE_SML_ENEMY = 4
#SHAPE_PLAYER = 6
#SHAPE_BULLETS = 16
#SHAPE_EXPLOSION = 20
#SHAPE_POWERUPS = 25

ASSET_ROOT$ = "Projects:resources/"  
ASSET_SHAPES$ = ASSET_ROOT$ + "shmup.iff"  
ASSET_STARS_BG$ = ASSET_ROOT$ + "shmup_stars_bg.iff"

DEFTYPE .w

Statement ASSET_LoadPalette{pal}  
  SHARED ASSET_SHAPES$
  LoadPalette pal,ASSET_SHAPES$
End Statement

Statement ASSET_LoadStarsBg{destBmp}  
  SHARED ASSET_STARS_BG$
  BitMap destBmp,320,512,5
  LoadBitMap destBmp,ASSET_STARS_BG$
  Scroll 0,0,320,256,0,256,destBmp
End Statement

Statement ASSET_LoadShapes{}  
  SHARED ASSET_SHAPES$
  BitMap 0,192,256,5
  LoadBitMap bmp,ASSET_SHAPES$

  GetaShape #SHAPE_LRG_ENEMY,0,0,32,32
  GetaShape #SHAPE_LRG_ENEMY+1,32,0,32,32

  GetaShape #SHAPE_MED_ENEMY,64,0,32,16
  GetaShape #SHAPE_MED_ENEMY,96,0,32,16

  GetaShape #SHAPE_SML_ENEMY,64,16,16,16
  GetaShape #SHAPE_SML_ENEMY,80,16,16,16

  For i=0 To 4
    GetaShape #SHAPE_PLAYER+i,0+(i*16),32,16,24
    GetaShape #SHAPE_PLAYER+i+5,0+(i*16),56,16,24
  Next i

  GetaShape #SHAPE_BULLETS,80,32,16,16
  GetaShape #SHAPE_BULLETS+1,96,32,16,16
  GetaShape #SHAPE_BULLETS+2,80,48,16,16
  GetaShape #SHAPE_BULLETS+3,96,48,16,16

  For i=0 To 5
    GetaShape #SHAPE_EXPLOSION+i,0+(i*16),80,16,16
  Next i

  GetaShape #SHAPE_POWERUPS,80,64,16,16
  GetaShape #SHAPE_POWERUPS+1,96,64,16,16
  GetaShape #SHAPE_POWERUPS+2,80,80,16,16
  GetaShape #SHAPE_POWERUPS+3,96,80,16,16

  Free BitMap 0
End Statement

Statement ASSET_Load{}  
  NPrint "Loading palette ..."
  ASSET_LoadPalette{#PAL_GAME}
  NPrint "Loading stars background ..."
  ASSET_LoadStarsBg{#BMP_PLAYFIELD}
  NPrint "Loading shapes ..."
  ASSET_LoadShapes{}
  NPrint "Done loading assets!"
End Statement

; -------------------------------------------------------------------

Statement GAME_Init{}  
  VWait 120
  BLITZ

  Buffer #BMP_PLAYFIELD,16384

  BitMap #BMP_STATUSBAR,320,#STATUSBAR_HEIGHT,5
  Slice #SLICE_STATUSBAR,44,320,#STATUSBAR_HEIGHT,$fff8,5,8,32,320,320
  Use Palette #PAL_GAME

  Slice #SLICE_PLAYFIELD,44+#STATUSBAR_HEIGHT,320,#PLAYFIELD_HEIGHT,$fff8,5,8,32,320,320
  Use Palette #PAL_GAME
End Statement

Statement DRAW_StatusBar{}  
  Use Slice #SLICE_STATUSBAR
  Use BitMap #BMP_STATUSBAR
  BitMapOutput #BMP_STATUSBAR

  Cls 14
  Print "TODO: Awesome statusbar here"

  Show #BMP_STATUSBAR
End Statement

Statement DRAW_Playfield{}  
  SHARED scrollY, playerX.q, playerTiltOffset, animateOffset
  Use BitMap #BMP_PLAYFIELD
  Use Slice #SLICE_PLAYFIELD

  UnBuffer #BMP_PLAYFIELD

  Show #BMP_PLAYFIELD,0,scrollY

  tile = #SHAPE_PLAYER+2+playerTiltOffset+(animateOffset*5)
  y = #PLAYFIELD_HEIGHT-32+scrollY
  BBlit #BMP_PLAYFIELD,tile,playerX,y

End Statement

;
; -------------------------------------------------------------------
;

ASSET_Load{}

GAME_Init{}

animateFrame = 0  
animateOffset = 0

playerTiltOffset = 0  
playerHorizAccel.q = 0  
playerX.q = 320/2-32/2

scrollY = 0

DRAW_StatusBar{}  
DRAW_Playfield{}

While NOT RawStatus(#KEY_ESCAPE)

  scrollY = QWrap(scrollY - #SPEED, 0, 256)

  animateFrame = QWrap(animateFrame+1, 0, 5)
  If animateFrame = 0
    animateOffset = QWrap(animateOffset+1, 0, 2)
  EndIf

  If RawStatus(#KEY_LEFT)
    playerHorizAccel - 1
  EndIf
  If RawStatus(#KEY_RIGHT)
    playerHorizAccel + 1
  EndIf
  playerHorizAccel = playerHorizAccel * 0.7
  playerX = QLimit(playerX + playerHorizAccel, 0, 320-16)
  playerTiltOffset = Int(playerHorizAccel)

  VWait
  DRAW_Playfield{}

Wend

End  

So yeah, very basic and/or crude for now. It doesn't look very impressive running yet either:

All you can do is fly the ship back and forth (but hey, it animates and the ship tilts a bit from side-to-side as you move left and right... exciting, right!?). I'm using the art assets from this package on OpenGameArt. Packaged everything together with a separate simple star background and exported as an IFF.

Actually the asset management has been a slight bit of a pain so far. I first noticed that Photoshop has the ability to save as Amiga IFF, so I would pre-convert the image I was working with to indexed colour mode and set up a 32 colour table and then save as an IFF. This seems to produce something that always is loadable by Blitz Basic (and AMOS for that matter), but occasionally produces a file that Deluxe Paint says is "mangled." Also sometimes, when loaded in Blitz Basic, the palette is somewhat off. Some colours will be totally different, but most will be correct. In this case, I've noticed I can fix the problem by opening the IFF in Deluxe Paint, and if it doesn't complain that it's mangled, I can simply re-save the file and the problem goes away. My suspicion is that Photoshop saves the palette information in a different way then what Blitz Basic is expecting, but I have not investigated this fully yet. I also tried using XnConvert by exporting from Photoshop as a indexed colour PNG or GIF and then converting to Amiga IFF with XnConvert, but still have run into the same issue (I think? I might also have mixed up my working files, heh, so will need to confirm that first I guess). At any rate, I definitely still have lots to do to nail down my asset preparation process.

Next steps will be putting together some custom font routines (I'm not sure how to generate the "Blitzfonts" that Blitz Basic supports, but at any rate, I don't think that supports bitmap fonts which is what I really want to use anyway). Following that will be adding the ability for the player to shoot and then adding enemies. I'm really curious to see how quickly I run into slowdowns.

Initial-ish Impressions: AMOS Pro vs Blitz Basic 2

I find the plethora of programming languages available for the Amiga fascinating. Especially so, the large amount of BASIC variants. The top two options for games specifically are definitely AMOS and Blitz Basic from what I can see.

Some other options such as HiSoft BASIC also seem to be of good quality (HiSoft BASIC actually does interest me quite a bit as it was advertised as being QuickBASIC compatible while not being mostly shit like Amiga BASIC from what I understand... but I will explore it later). However, it seems most people at the time settled with either AMOS or Blitz.

I've personally spent more time with AMOS so far, but this weekend I spent a bit of time with Blitz Basic and figured I'd jot down my initial thoughts about each.

None of the points below are listed in any particular order, and it is all just based on my impressions, opinions and experiences from only a limited amount of time looking at each thus far. Also note that the below screenshots show my own customizations, they are not how the "out of the box" experience is exactly.

AMOS Pro

  • Editor looks very polished. Especially as compared to the earlier (non-pro) versions. Lots of customization available with the ability to load "accessories", which, due to AMOS's overwhelming popularity back then, there are a great many to choose from.
  • The editor is not without it's quirks. For one I dislike the automatic formatting (which you cannot turn off, and is likely a result of the fact that the editor is obviously tokenizing your code as you type it in). More specifically, I don't like how it removes excess whitespace and uppercases variable names, e.g. if I type ship_x = foo + bar, it will always auto-format to SHIP_X=FOO+BAR.
  • AMOS Pro thankfully added INCLUDE support for modularizing your code, but even so, you'll probably end up with big source files. Luckily the editor allows you to collapse Procedures (though you might not actually want to use them in all cases, since this is a platform where the performance difference between GOSUB and a Procedure call can really matter!)
  • Following up on the above mentioned "accessories", the developer of AMOS provided some that serve as tools for sprite/image editors, map editors, font editors, and more. These are easily accessible from within the IDE.
  • Lots of documentation in the forms of a beefy user manual and many other published books and sample code. Even built in fully-browseable help within the editor itself.
  • Easily accessible "direct mode" available from the editor to allow quick experimentation (it also doubles as a debugging tool when your program encounters an error).
  • Very extensive set of functions with what looks like a decent abstraction over the underlying hardware (from my early impressions anyway). I personally think this is a pro and a con (a pro for when you're starting out as it's easier to jump into and a con because I suspect it won't help you that much if you move onto e.g. C/assembler later on). This abstraction covers things like asset loading as well which is certainly convenient.
  • The AMOS BASIC dialect is overall good, though lacking in some key areas. For example, no SELECT and TYPE. Somewhat odd syntax for calling Procedures (SUBs and FUNCTIONs from QBasic) and accessing the return value (if any). You cannot put comments just anywhere (well, unless you use REM I guess, though in my opinion, REM makes the code a bit harder to read). A comment written using ' must be on it's own line.
  • Quirky / annoying support for accessing system libraries, but it is somewhat workable.
  • Doesn't integrate with the OS, kind of does it's own thing with it's own totally custom GUI. This also extends to programs developed with it if they do things like pop up file requestors which will be quite alien-looking.

Blitz Basic 2

  • Much more powerful language that is clearly built to enable high performance as a primary goal. Blitz's BASIC dialect is excellent for the time in my opinion. Support for TYPE's, pointers, inline assembly, and more.
  • The language is built for a compiler and there doesn't appear to be any interpreter at all (even when running from the IDE, you can't just choose "Run" unless you've already compiled your program). Thus performance is higher from the get-go.
  • The editor is minimal, which is nice. Though it definitely is unique in a number of ways, and I don't mean that in a good way, at least not if you like me are accustomed to modern text editor standards. You cannot always hit Return to insert a new line (only if you're at the end of the current line), else you need to press Amiga+I to insert a new line or hit Shift-Return. Using the Backspace or Delete keys to remove lines is not possible, you instead must use a menu command (or press Amiga+D) to delete lines. Selecting text for copy+paste is kind of quirky, requiring you to select the text to be copied then go and find the location where you want it copied to (the paste location) and place the cursor there (moving the cursor does not unselect text) and then issue the menu copy command (or Amiga+C). There is a specific menu command for unselecting text. I'm sure you can get used to all this, but I point it out only as it's pretty different from what most people would be used to.
  • The editor has syntax highlighting! Only two colours though: regular text and tokens.
  • The editor has a side pane that pops up when your program has labels (for GOTO/GOSUB only, doesn't show Statement or Function names for some weird reason). Clicking the names in this pane scrolls immediately to that label. Thusly you can also use this as a "bookmark" list of sorts.
  • The editor is incredibly unstable. It crashes a lot, in a lot of different situations, forcing you to do a Ctrl-Amiga-Amiga reset. I've had the editor crash when clicking Project -> New, when switching back to the Workbench, when compiling a program, many times when debugging code with direct mode (in all cases, this was incredibly simple code, not even using "blitz mode"). Worth noting, I've tried four different versions of Blitz2, the 1.6 Amiga Format disk release, a 1.7 release, 2.0 and 2.1. I actually also tried installing Workbench 3 temporarily just so I could try the updated Ted version and also the "SuperTed" update, hoping that these would improve things. However, all of these also were incredibly unstable for me.
  • The standard library appears to be a bit "closer to the metal" then AMOS, but it still abstracts the hardware quite a bit. I suspect that moving from Blitz Basic to C/assembly would be a bit easier. In fact, the mere ability to use inline assembly with such ease basically anywhere in your code means that you possibly wouldn't even need to switch to C or go full-out with pure assembly unless your performance requirements were quite extreme.
  • The editor has built in help which is either limited to parameter lists only, or in later versions even has a browseable function index with descriptions and (few) examples.
  • There is not nearly as much documentation out there for Blitz Basic as there is for AMOS. A common complaint about Blitz Basic from what I've read is that it's harder to get started with, though I think for an experienced programmer that this difference is moot. Probably the smaller amount of example code that's out there would be the biggest annoyance.
  • The debugger and direct mode support looks a fair bit more unpolished than AMOS at first glance, but is definitely more advanced. Especially with a debugger update that is available for release 2.1 (though coincidentally I found this debugger update appeared to reduce stability somewhat).