Driver testing, USBKEYB, USBDRIVES, TSRs

[c.m]

Hi Bret,

I have tested the latest USB drivers (from 2009-09-13) on my DOS computer now. The computer has two UHCI controllers, the first controlling the two rear USB connector and the second controlling two internal connectors. The BIOS has an option for mouse/keyboard "legacy support" but that never worked (showed error message during POST, didn't load DOS) so it's turned off. It might be important that I always used JemmEx during the testing. Here's what I noted:

• Using both USBKEYB and an Int15.4F based German keyboard layout driver ({J}KEYBGR from Japheth, the J one operates as JLM in Protected Mode, the other is a simple Real Mode TSR) types any "translated" key twice. Other layout drivers operating different work how they're supposed to (i.e. they translate the layout for all keyboards). With this bug, the key is still translated but gets to DOS twice. The bug even affects the PS/2-connected keyboard and is in effect when no USB keyboard is connected, too. USBKEYB uses Method 1 for emulating keystrokes from USB keyboards.
  
• I previously installed USBUHCI twice (for both controllers) and then went to uninstall it. I had to manually specify to uninstall the second instance first, otherwise the uninstaller found the first instance and complained that it couldn't uninstall it because of the interrupts hooked by the second instance. This raises two questions: Why doesn't the uninstaller search installed HCI drivers backward (avoiding this particular problem), and why can't it unhook the interrupt with another HCI driver instance on the same interrupt? Shouldn't this work with the interrupt sharing protocol (IISP)?
  
• I connected an 'internal' card reader to the two internal connectors (i.e. the second controller) which has two slots both for various card formats. USBDRIVES correctly found the device and assigned a DOS drive to the SD slot when I inserted a SD card. However, the access was very slow. Accessing a single sector with the WDE sector editor required a few seconds each time, even when only accessing the same two sectors (boot sector and first FAT sector) again and again. Executing "DIR" was slow too and stopped at the summary some minutes when computing the number of free clusters (it has to search all ~ 256 FAT sectors of the FAT16 drive). Executing "DIR" a second time was as slow, but notably the summary again took minutes. DOS usually buffers the number of free clusters in the DPB even for FAT16 drives, especially if no cluster allocation or deallocation happens inbetween. I uninstalled USBDRIVES and reinstalled with some other X settings. This didn't help the performance much. Single sector reads were definitively as slow as before. DOS usually only does single sector reads except maybe when reading via Int25/26 or from a file to the user buffer via Int21.3F. Notably, the card reader (or the card?) didn't like X settings 4 and 8 at first, but after retrying to read the first sector a few times, it accepted all further requests (and returned valid data). USBDRIVE S always showed the capacity under "DISK/LUN" as zero (Total Sectors being zero too), even when the card was detected and correctly assigned a DOS drive.

These are some random notes that aren't related to the mentioned testing, but that I'd like to write you anyway:

• Are your TSRs compatible with a DOS that moves the DOS data segment? (This really isn't necessary, I'm just interested.)
  
• You apparently use Int2F for dynamically (?) allocated multiplex numbers. Not to say this doesn't work, but it potentially could cause problems with other programs. Did you think about using AMIS? (It specifies the alternative multiplex interrupt as 2Dh. The most recent specification is completely in RBIL.)
  
• You can extend your FindInt code by two methods to find the interrupt handler in the chain: Add the 'uninstalled' IISP headers, which is created by iHPFS (simply 90h,EAh (nop, then jump far to the next handler) and the "KB" signature) on some (soft) Ints when uninstalling, and search through the interrupt lists provided by installed AMIS TSRs. The AMIS interrupt hook list might provide you another entry to the interrupt chain when it's blocked by an incompatible TSR installed later.
  
• Why do you prefer A86 over alternatives? I don't want to tell you what you 'should' use, again I'm just interested. (However, if you want to discuss assemblers..)
  
• Probably not many of the USB programs access files anyway, but do you support long file names? At least USBPRINT's COPY works with user-specified files, so LFN support might be useful to some.
  
• You have to close all handles (usually the Std handles 0..4) before issuing Int21.31; otherwise they'll stay open afterwards. This isn't critical for the default CON handle (because that's shared by all processes anyway) but could, for example, hold the destination file of an output redirection blocked.
  
• When a program installs resident and doesn't need to use files afterwards, its PSP can be deallocated. This is very easy to do for your auto-relocation code.
  
• Your memory allocation code is good, but still might create a memory hole when the environment is smaller than the resident code to install and there's no (or not enough free) UMA. Best results can be obtained by relocating the whole process to the top of the LMA (low memory area) before installing the resident part, then freeing the environment (as you already do), and then allocating a memory block for the resident part. (I initially wrote a lot more code for my TSRs to determine whether the process relocation would be useful, but after all, it's much simpler just to relocate the process always.)
  
• You currently leave some unnecessary bytes in memory that aren't used for anything. This is the description of the program at the start of the executable plus over half of the things in the PSP which DOS doesn't require for the process, such as the command line. Generally, all PSP fields from address 50h can be re-used for your own data or code.
  
  Edit: Wait, the description is used by the multiplex interface, isn't it?

Regards,
Christian

[Bret]

I have tested the latest USB drivers (from 2009-09-13) on my DOS computer now.

Glad you took the time to do it and respond in detail!

It might be important that I always used JemmEx during the testing.

It could, though I don't suspect it makes much difference. The main reason I don't use JEMM is because it doesn't support I/O virtualization, so it doesn't work correctly with USBJSTIK.

Using both USBKEYB and an Int15.4F based German keyboard layout driver ...

I'm not sure what you're saying here. Are you saying Japheth's drivers don't work but others do? I will do some testing when I get some time to see what's happening.

I can tell you, however, that INT 15.4F based keyboard translators in general don't work very well, and I don't recommend using them. They almost never translate properly any of the modifier (shift, alt, control, windows), lock (caps, scroll, num), "special" (printScreen, break, etc.), or multimedia keys. They also don't work with all programs, because some programs that do their own keyboard processing don't call INT 15/4F. You're usually better off with a "real" keyboard translator like KEYB, even though it uses more memory. In addition, FreeDOS KEYB is NOT equivalent to MS-DOS KEYB, and doesn't work in some situations where MS-DOS KEYB does.

I previously installed USBUHCI twice (for both controllers) ...

The reason it won't automatically uninstall things in reverse order is modularity. For instance, if you have four instances of USBUHCI{L} installed, but only want to uninstall the second one, you can do that. The only one you can't uninstall until all of the others are uninstalled is the first one, but it doesn't have anything to do with Interrupts. The first instance contains all of the program code, common data that ALL instances need (including Device 0 registration data for the individual device drivers), as well as data for the individual host controller it is managing. Instances after the first one only contain the data for the specific host they are managing. That's why instances after the first one require so much less memry.

I have actually considered dividing the program into two separate modules, a "common module" and "data-only module", so that you could just leave the common module in memory and install/uninstall individual host data modules as required. I may in fact do that some time in the future, but it is not that way now. So, you will usually want to make the first instance manage the host controller you use most often and wouldn't normally want to uninstall -- in your case, I suspect the one associated with the external ports, since those would control your USB mouse and keyboard and flash drives.

I connected an 'internal' card reader ...

For a USB drive with removable media (such as a multimedia card reader), USBDRIVE has to treat it the same way it (and DOS) treats a floppy drive. There's a two second timer that starts when the disk is accessed, and when the timer expires, the next time the disk is accessed USBDRIVE returns an "I don't know if the media has changed or not" to DOS. DOS, of course, treats this like a "Media has changed". USBDRIVE can't tell if media has changed, because USB multimedia card readers usually don't come with media change lines (they should, but they don't).

Also, with card readers, USBDRIVE must keep polling the empty physical slots (in the background, of course) to see if media has been inserted so it can read partitions and assign drive letters. Some card readers take a VERY long time to respond to those polls if there's no media (many, many seconds), and none of the other slots (even ones with media in them) are accessed at the same time. I think it might be possible to be doing things wth all slots at the same time, but haven't tried it. It would increase TSR memory requirements, and some devices may not like it anyway.

It's also necessary to retry every operation multiple times, especially with USB disks, since they are so flaky and inconsistent with their responses. For example, I have an internal RealTek card reader on my desktop that takes more than 30 seconds before USBDRIVE to even get to the fourth physical slot to see if there's media installed, and I have a cheap generic external media reader that does it all in a few seconds.

There are definitely some disadvantages to treating USB disks as removable hard drives instead of permanent hard drives like most of the other drivers do, especially on USB disks with removable media. There's still a LOT of work that could be done to USBDRIVE, especially regarding timing and compatibility issues. Frankly, I would LOVE for somebody to take USBDRIVE off my hands and turn it into the program it should be. We'll see if that ever happens, though.

Are your TSRs compatible with a DOS that moves the DOS data segment? (This really isn't necessary, I'm just interested.)

I'm not sure what you mean by that. As long as the DOS data the programs need to access (like the LOL, CDS, and DPB for USBDRIVE) is addressable with segment:offset, everything should work just fine. The USB programs generally use documented DOS and BIOS API calls, and don't directly access internal DOS structures or data areas unless they have to. Data that got moved to XMS or EMS could be a problem, though, unless it's accessible through an API.

You apparently use Int2F for dynamically (?) allocated multiplex numbers ...

I'm familiar with AMIS, and have considered using it. I think the dynamically allocated process for INT 2Fh works pretty well, and I've never had problems with it so far (at least not that anybody's ever told me about). In fact, I modeled the process after an early version of AMIS, but thought AMIS was too inefficient and did not follow the existing TSR conventions. If I did run into problems with the process I used now, I might start using AMIS. I personally think some of the fundamental premises of AMIS are flawed, since INT 2Fh has specifically been dedicated to TSR's from the beginning. I do agree that something like AMIS and IIRP should have been developed AND enforced by MS/IBM since the early 1980's, but they didn't and the result is kind of a mess. I don't think AMIS is the complete answer, though.

You can extend your FindInt code by two methods to find the interrupt handler in the chain: Add the 'uninstalled' IISP headers, which is created by iHPFS (simply 90h,EAh (nop, then jump far to the next handler) and the "KB" signature) on some (soft) Ints when uninstalling, and search through the interrupt lists provided by installed AMIS TSRs. The AMIS interrupt hook list might provide you another entry to the interrupt chain when it's blocked by an incompatible TSR installed later.

I'm not sure what you mean by all of this. Do you have some code (or at least pseudo-code) to explain it in more detail?

Why do you prefer A86 over alternatives? I don't want to tell you what you 'should' use, again I'm just interested. (However, if you want to discuss assemblers..)

I use A86 because it's the one I'm familiar with. I've been using it for a LONG time, since way before NASM and FASM even existed, and I've always despised MASM and TASM syntax. A86 also has a very nice symbolic debugger, which I don't think any of the other assemblers have for .COM programs (I could be wrong about that, though). From what I've seen, FASM and NASM both have some nice features, but nothing compelling enough to make me switch, at least for the kinds of programs that I write. If I didn't already have a licensed copy of A86, though, I definitely would feel different about the whole thing.

Probably not many of the USB programs access files anyway, but do you support long file names? At least USBPRINT's COPY works with user-specified files, so LFN support might be useful to some.

The programs do support LFN when it's appropriate. USBPRINT's COPY option does support LFN (as long as the underlying OS also supports it, of course).

You have to close all handles (usually the Std handles 0..4) before issuing Int21.31 ...

That is not my understanding. My understanding is that CON and ERR are always open, and can never be closed. I never open them before I write to them (so they must therefore always be open), nor do I close them (since I do not open them).

Are you essentially saying that DOS does not always clean up after itself if something goes wrong with an input or output redirection?

With the memory allocation method you discuss below, it would not be necesssary to call Int21.31 anyway.

When a program installs resident and doesn't need to use files afterwards, its PSP can be deallocated. This is very easy to do for your auto-relocation code.

The PSP can't be deallocated in a .COM program, at least not if you and I are thinking about the same kind of deallocation. The environment can be deallocated, and I do that. Portions of the PSP can be used as a data storage area for the TSR, though I don't currently do that (you talk about that down further).

Your memory allocation code is good, but still might create a memory hole ...

I've actually thought about doing that in the past, but have never actually tried to implement it. I don't think it wouldn't be very difficult -- I may try it in the near future. Do you have some sample code?

You currently leave some unnecessary bytes in memory that aren't used for anything ...

Yes, the description is used by the multiplex interface, in addition to handling things "properly" if the user TYPEs the .COM file to the screen. I have never tried to use the PSP to store data, just because it's a little bit of a pain to do -- you must be completely done with command-line processing and file operations before you can even initialize the data. I know it's possible, but have just never done it.

FWIW, in USBUHCI{L}, there's also some unused memory surrounding the Frame List (the basis of the USB Schedule). The beginning of the Frame List must be aligned on a 4k page boundary, so I have to reserve 8k of space for it and then wait until I know where the TSR is installed in memory before I can calculate where to put it. I then use parts of the remaining 4K (some before the Frame List and some after) for a TSR stack and various data tables, but don't end up using all of it.

Now that I've started working on the OHCI driver, I've also discovered that OHCI reserves a 4k memory page for part of the control process, but only uses 144 bytes of it, and it's impossible to use the remaining memory for anything else. Luckily, this particular memory is MMIO and uncachable, so is assigned way high in memory by the BIOS, and is not part of the TSR memory. Nonetheless, it's still a waste.

They definitely don't design things any more with efficient memory use in mind.

Bret

[c.m]

It could, though I don't suspect it makes much difference. The main reason I don't use JEMM is because it doesn't support I/O virtualization, so it doesn't work correctly with USBJSTIK.

Did you ask Japheth about that? I think I once read something from him regarding the virtualization interface, but I don't know exactly.

With the I/O virtualization support, couldn't there be a "Method 4" for USBKEYB (and your other program which emulates key strokes, SCANCODE?) to emulate the keyboard by hooking the associated ports?

Using both USBKEYB and an Int15.4F based German keyboard layout driver ...

I'm not sure what you're saying here. Are you saying Japheth's drivers don't work but others do?

Yes, exactly.

I can tell you, however, that INT 15.4F based keyboard translators in general don't work very well, and I don't recommend using them. They almost never translate properly any of the modifier (shift, alt, control, windows), lock (caps, scroll, num), "special" (printScreen, break, etc.), or multimedia keys.

KEYBGR seems to work well with all keys. Note that the translation to a German layout indeed has to translate some shifted keys, a few characters which are available with AltGr only, and the decimal dot on the num pad is a decimal comma instead.

They also don't work with all programs, because some programs that do their own keyboard processing don't call INT 15/4F.

I've tested that a bit, and found you're right. KEYBW (another simple German layout driver, hooking Int09) works with the mentioned WDE (full-screen) sector editor, while KEYBGR doesn't. How exactly does this work?

You're usually better off with a "real" keyboard translator like KEYB, even though it uses more memory. In addition, [FreeDOS KEYB] doesn't work in some situations where MS-DOS KEYB does.

Could you explain that? How (at what level) do FreeDOS and MS KEYB translate the keys?

The reason it won't automatically uninstall things in reverse order is modularity. For instance, if you have four instances of USBUHCI{L} installed, but only want to uninstall the second one, you can do that. The only one you can't uninstall until all of the others are uninstalled is the first one, but it doesn't have anything to do with Interrupts.

It displayed the following message, so I thought it was just caused by the hooked interrupt. You could probably add another error message to inform the user that the instance with the code and common data (i.e. the first instance) can't be uninstalled as long as there are other instances.

Code: Select all

The IRQ being used by USBUHCI is being "shared" by another program
  that was installed into memory after USBUHCI was.
Most likely, that other program is also a USB-related program.
You will need to Uninstall the other program from memory first,
  and then come back and try to Uninstall this one again.

I connected an 'internal' card reader ...

For a USB drive with removable media (such as a multimedia card reader), USBDRIVE has to treat it the same way it (and DOS) treats a floppy drive. There's a two second timer that starts when the disk is accessed, and when the timer expires, the next time the disk is accessed USBDRIVE returns an "I don't know if the media has changed or not" to DOS. DOS, of course, treats this like a "Media has changed". USBDRIVE can't tell if media has changed, because USB multimedia card readers usually don't come with media change lines (they should, but they don't).

Okay, that explains why DOS recomputed the free cluster count.

Also, with card readers, USBDRIVE must keep polling the empty physical slots (in the background, of course) to see if media has been inserted so it can read partitions and assign drive letters. Some card readers take a VERY long time to respond to those polls if there's no media (many, many seconds), and none of the other slots (even ones with media in them) are accessed at the same time. I think it might be possible to be doing things wth all slots at the same time, but haven't tried it.

I noticed that sometimes USBDRIVE S displayed the empty slots (one or both) as "Initializing" instead of showing the fake drive data with Total Sectors and Sector Size as zero. So that's probably my problem here. It might interest you that in my (Windows) PC I've the same internal card reader, and Windows works just fine with it (and the same card I tested). This probably indicates that Windows does indeed access the slots at the same time.

It would increase TSR memory requirements,

You could add it as build conditional only, i.e. provide different binaries of USBDRIVE.

Are your TSRs compatible with a DOS that moves the DOS data segment? (This really isn't necessary, I'm just interested.)

I'm not sure what you mean by that. As long as the DOS data the programs need to access (like the LOL, CDS, and DPB for USBDRIVE) is addressable with segment:offset, everything should work just fine. The USB programs generally use documented DOS and BIOS API calls, and don't directly access internal DOS structures or data areas unless they have to. Data that got moved to XMS or EMS could be a problem, though, unless it's accessible through an API.

No, I mean a DOS where its main data segment (containing the List of Lists and SDA) is relocated to UMA (or back to LMA). CDS and DPBs are potentially relocated later, too. However, anything remains in the first MiB so it's still addressable as usually. You're allocating the DPBs in your memory, so external software can't relocate them anyway. Note that the DOS data (either main segment, or other stuff like the CDS) is usually relocated during CONFIG.SYS processing only, so you don't have to care about that.

You can extend your FindInt code by two methods to find the interrupt handler in the chain: Add the 'uninstalled' IISP headers, which is created by iHPFS (simply 90h,EAh (nop, then jump far to the next handler) and the "KB" signature) on some (soft) Ints when uninstalling, and search through the interrupt lists provided by installed AMIS TSRs. The AMIS interrupt hook list might provide you another entry to the interrupt chain when it's blocked by an incompatible TSR installed later.

I'm not sure what you mean by all of this. Do you have some code (or at least pseudo-code) to explain it in more detail?

It's a bit of a mess but attached is the code I'm using in my TSRs. (The forum expands tabs another way than I intend it to, so I can't use code tags. I'm using tab size 8.) Note that I added the .harder attempt later which implements searching through the interrupt lists of installed AMIS TSRs. The searchiispchain code also allows the "unused IISP" header. Note that I don't verify all of the values for the "unused IISP" header.

Why do you prefer A86 over alternatives? I don't want to tell you what you 'should' use, again I'm just interested. (However, if you want to discuss assemblers..)

I use A86 because it's the one I'm familiar with. I've been using it for a LONG time, since way before NASM and FASM even existed, and I've always despised MASM and TASM syntax.

I think MASM's syntax isn't so different from A86. The (MASM 5) RxDOS sources where ported to A86 by someone, but it didn't take much effort and most of the actual code was unchanged.

A86 also has a very nice symbolic debugger, which I don't think any of the other assemblers have for .COM programs.

Probably not, but then again I'm not missing it. I use (a slightly modified version of) FD DEBUG for debugging, and found that the map files optionally created by NASM are very useful for this. Or just reading the source along debugging in another window or on another machine, but that's probably cheating ;-)

From what I've seen, FASM and NASM both have some nice features, but nothing compelling enough to make me switch, at least for the kinds of programs that I write.

If it wasn't for it being my first assembler, I would chose NASM any time again for the multi-section bin output format. It allows a lot of complex segment setups and sectioning your binary with different "virtual" start offsets, which either doesn't work with the object format or I'm just to stupid to use it. I feel like FASM could do some of these things too (and some other things NASM can't do), but it isn't documented well enough to learn these.

You have to close all handles (usually the Std handles 0..4) before issuing Int21.31 ...

That is not my understanding. My understanding is that CON and ERR are always open, and can never be closed. I never open them before I write to them (so they must therefore always be open), nor do I close them (since I do not open them).

Are you essentially saying that DOS does not always clean up after itself if something goes wrong with an input or output redirection?

No, I'm saying that DOS never closes any handles when you "go TSR" with Int21.31 or Int27. This has always been that way, and DOS's behaviour mustn't change now that some TSRs might rely on using the Std or other handles after they've terminated. This causes two problems: When redirecting to a device (such as NUL), the SFT will stay used and can't be freed. Some people don't think that's an issue, but I feel like it's one. SFTs are a limited resource. Installing and removing a TSR again and again, while redirecting the output of the installation to NUL each time, eventually causes the system to run out of SFTs. The second problem is that when the output is redirected to a file, the SFT stays open. In an environment without proper file sharing, the file won't be affected by that, but the used directory entry (which, after removal of the file, might be re-used for another file) and FAT cluster chain could be corrupted. In an environment with proper file sharing, the file will be blocked and can't be deleted.

It's also technically wrong to state the Std handles were never opened. They're opened by the DOS installation code initially (to the standard devices CON, AUX, PRN) or by the shell for redirection. They're then duplicated to a child process when it is created (usually by executing a program), similar to the duplication done by Int21.45, "DUP". The system file table (SFT) entry for the file remains open until all handles (the original one plus any duplicate) are closed. Closing of the Std handles happens along with closing any files left open in the DOS process termination handling (Int21.4C). TSR termination doesn't close any files.

When a program installs resident and doesn't need to use files afterwards, its PSP can be deallocated. This is very easy to do for your auto-relocation code.

The PSP can't be deallocated in a .COM program, at least not if you and I are thinking about the same kind of deallocation.

It's PSP is deallocated by the usual process termination (21.4C). As discussed below, since we can't assume there's enough free UMA for the TSR, we have to relocate the PSP first and then install our TSR, (possibly) where the PSP previously was located. (Actually, the user might even think it's a good idea to "load high" your process. This might simply block the UMA required for the TSR, or will otherwise create memory fragmentation with your current method. I'd say it's sufficient to state in your documentation that the user shouldn't load the process "high", but with the relocation it's possible to fix this when the program runs.)

Your memory allocation code is good, but still might create a memory hole ...

I've actually thought about doing that in the past, but have never actually tried to implement it. I don't think it wouldn't be very difficult -- I may try it in the near future. Do you have some sample code?

Currently I've just the "complicated" code which first allocates a block, then determines (by MCB walking, which isn't optimal) whether PSP relocation would be useful, and relocates the PSP afterwards to re-allocate the block. Either you want to read that too, or I might get around to fix my code in the next days.

You currently leave some unnecessary bytes in memory that aren't used for anything ...

Yes, the description is used by the multiplex interface, in addition to handling things "properly" if the user TYPEs the .COM file to the screen.

I noticed this, but this cause isn't related to the resident part of the TSR.

I have never tried to use the PSP to store data, just because it's a little bit of a pain to do -- you must be completely done with command-line processing and file operations before you can even initialize the data. I know it's possible, but have just never done it.

I think it's best to do command-line processing first anyway. However, file processing still works even when re-using bytes 50h+ of the PSP.

FWIW, in USBUHCI{L}, there's also some unused memory surrounding the Frame List (the basis of the USB Schedule). The beginning of the Frame List must be aligned on a 4k page boundary, so I have to reserve 8k of space for it and then wait until I know where the TSR is installed in memory before I can calculate where to put it. I then use parts of the remaining 4K (some before the Frame List and some after) for a TSR stack and various data tables, but don't end up using all of it.

Well, you really can't do more about this than re-using some of the "wasted" memory.

[Bret]

Did you ask Japheth about that? I think I once read something from him regarding the virtualization interface, but I don't know exactly.

I e-mailed him, but he never responded. At the same time, I also asked him about creating a real-mode VxD interface similar to what Windows provides, so that a real mode program could use the services that now require a JLM (one of which is I/O virtualization).

With the I/O virtualization support, couldn't there be a "Method 4" for USBKEYB (and your other program which emulates key strokes, SCANCODE?) to emulate the keyboard by hooking the associated ports?

Not with the standard I/O virtualization interface as implemented by MS and Qualitas, since it only works with I/O ports > 100h, and the keyboard (and PS2 mouse) use ports 60h-64h. If the VxD interface was extended to real mode, though, that might be possible. I don't know if there's a legitimate technical reason for the port number limitation or not. I/O virtualization will also be required to write USB serial port and sound card drivers, and may be other types of hardware emulators as well.

Could you explain that? How (at what level) do FreeDOS and MS KEYB translate the keys?

When a program accesses the keyboard directly, it intercepts INT 09. When an INT 09 occurs, the software reads I/O port 60h to determine the scan code that was pressed. Then, if the program is "compatible" with what IBM says it is supposed to do, it issues an Int15.4F to "translate" the scan code into a new one. In then processes the scan code using the "translated" value.

There are several problems with this, the first being that a single key press (or release) can create multiple scan codes. So, the keyboard driver must "remember" the last several keys that were pressed to know how to correctly translate the current one. There are also certain cases where the current scan code cannot be translated correctly until the NEXT scan code is received, which is impossible to do in a simple Int15.4F handler.

There are also some keyboards that have different "modes" they can be in, for instance a latin mode and a greek or cyrillic mode (german keyboards wouldn't need this). The keyboard driver and the user must "agree" about which mode is currently being used. The keyboard driver must also know which code page is currently in use, as must the program itself in some cases.

Keyboards themselves also send out different scan codes depending on the states of certain modifier keys. For instance, the scan codes that get sent when you press the PrtScr/SysRq key depend on which combinations of Shift, Alt, AltGr, Control, and Windows keys you have pressed at the same time. This "translation" actually happens INSIDE the keyboard, so the keyboard driver must be "in sync" with what the keyboard hardware is doing and what the user thinks should be happening.

There are various other obscure idiosynchrocies that can occur in a keyboard driver, which a simple Int15.4F driver just can't handle properly, even if the program calls it. Keyboard processing is MUCH more complicated than most people realize.

There is a keyboard driver built into the BIOS, which is usually just set up for a US keyboard. In simple terms, the MS-DOS KEYB program replaces the ENTIRE BIOS keyboard driver with a new one, so nothing in the original BIOS driver is used any more. FreeDOS KEYB and Int15.4F programs do not replace the entire BIOS driver, but only replace certain pieces of it .

The IRQ being used by USBUHCI is being "shared" by another program that was installed into memory after USBUHCI was.

This particular error message might indicate that you installed one drive as USBUHCI and the other with USBUHCIL. Although the two programs are related, they don't "know" about each other and treat each other as if they were made by somebody else.

It might interest you that in my (Windows) PC I've the same internal card reader, and Windows works just fine with it (and the same card I tested). This probably indicates that Windows does indeed access the slots at the same time.

Yes, the same with some of mine -- Windows seems to do a better job with that. I think USBDRIVE could do the same thing, but it would take more development time and more memory to do it. At least USBDRIVE works, which was my goal, even though it's a far cry from optimal.

TSR termination doesn't close any files.

I never knew that, and will fix things in future versions of my programs (actually, I'll probably implement the memory relocation scheme you talk about which doesn't "go TSR"). But, I still stand by my original statement that DOS isn't cleaning up after itself like it should. The fact that a program terminates as a TSR rather than just terminating shouldn't make any difference as to how DOS handles its own open files. Like you said, it's too late to change it now, but it probably shouldn't have been that way in the first place (but maybe there's a good reason for it that just isn't obvious).

Either you want to read that too, or I might get around to fix my code in the next days.

I'll look at what you have, but if you're going to be fixing the code anyway, send me a copy when you're done.

However, file processing still works even when re-using bytes 50h+ of the PSP.

Actually, that's only true if you move the DTA first. I ran into problems once when I tried to open a file while still processing the command line (testing a file name provided on the command line for validity) and it destroyed the command line. The default DTA is PSP:80h, the same place the command-line parameters are located.

Actually, after the memory has been moved around and the program terminated, does any part of the PSP really need to be preserved any more (even the stuff from 0h-50h)? The PSP might actually be a good place to put the TSR stack, especially if you could have the whole 256 (0-FFh) bytes instead of just 176 (50h-FFh). 176 might be enough, but 256 is definitely better -- best to not be skimpy with stack space.

[japheth]

Why do you prefer A86 over alternatives? I don't want to tell you what you 'should' use, again I'm just interested. (However, if you want to discuss assemblers..)

I use A86 because it's the one I'm familiar with. I've been using it for a LONG time, since way before NASM and FASM even existed, and I've always despised MASM and TASM syntax.

I think MASM's syntax isn't so different from A86. The (MASM 5) RxDOS sources where ported to A86 by someone, but it didn't take much effort and most of the actual code was unchanged.

I did a few tests assembling an usbdos source file (drives.a06) with jwasm. In this test drive.a06 was included in a "master" file which does some housekeeping and makes jwasm more A86 friendly by "redefining" some instructions and directives (so it can digest the '>' after a conditional jump). I still got about 50 errors in total, and in detail it was:

- ca 35 errors because of "duplicate symbols" ( S20, P80, ... )
- 5 errors because of "MOV seg_reg, seg_reg" instruction
- 1 error because of AAM <number>
- 1 error because of DUP ' ' (masm syntax requires 2. operand of DUP in brackets )
- 2 errors because of dword numbers with a '_' "separator"
- 2 errors because of size mismatch
- 2 errors because of INC CX,2
- 1 error because of TEST [87h],8 (masm syntax wants a DS: before the [87h])

So it's not too difficult to make the source compatible with JWasm. The only true show stopper are the "duplicates".

[c.m]

Did you ask Japheth about that? I think I once read something from him regarding the virtualization interface, but I don't know exactly.

I e-mailed him, but he never responded. At the same time, I also asked him about creating a real-mode VxD interface similar to what Windows provides, so that a real mode program could use the services that now require a JLM (one of which is I/O virtualization).

Write your suggestions now. I'd be interested in that VxD interface too. (Theoretically, your driver is able to provide that itself. The JLM still needs to be loaded with the JLM load program, however.)

With the I/O virtualization support, couldn't there be a "Method 4" for USBKEYB (and your other program which emulates key strokes, SCANCODE?) to emulate the keyboard by hooking the associated ports?

Not with the standard I/O virtualization interface as implemented by MS and Qualitas, since it only works with I/O ports > 100h, and the keyboard (and PS2 mouse) use ports 60h-64h. If the VxD interface was extended to real mode, though, that might be possible. I don't know if there's a legitimate technical reason for the port number limitation or not.

Probably not, as for the other related limitations/bugs.

I/O virtualization will also be required to write USB serial port and sound card drivers, and may be other types of hardware emulators as well.

Yeah, people are talking about sound card drivers (not for USB sound cards, but for normal PCI ones) and I/O virtualization sometimes.

Could you explain that? How (at what level) do FreeDOS and MS KEYB translate the keys?

When a program accesses the keyboard directly, it intercepts INT 09. When an INT 09 occurs, the software reads I/O port 60h to determine the scan code that was pressed. [...]

There is a keyboard driver built into the BIOS, which is usually just set up for a US keyboard. In simple terms, the MS-DOS KEYB program replaces the ENTIRE BIOS keyboard driver with a new one, so nothing in the original BIOS driver is used any more. FreeDOS KEYB and Int15.4F programs do not replace the entire BIOS driver, but only replace certain pieces of it.

The replaced parts of the BIOS however do reside in Int09 (and possibly Int16), don't they? This means that programs which hook Int09 (not passing the request to underlying Int09 handlers) don't use the translated layout anyway.

The IRQ being used by USBUHCI is being "shared" by another program that was installed into memory after USBUHCI was.

This particular error message might indicate that you installed one drive as USBUHCI and the other with USBUHCIL. Although the two programs are related, they don't "know" about each other and treat each other as if they were made by somebody else.

Nope, I installed two USBUHCI instances. The second one also uses less memory than the first one, so the TSRs have to know about each other.

TSR termination doesn't close any files.

I never knew that, and will fix things in future versions of my programs (actually, I'll probably implement the memory relocation scheme you talk about which doesn't "go TSR"). But, I still stand by my original statement that DOS isn't cleaning up after itself like it should. The fact that a program terminates as a TSR rather than just terminating shouldn't make any difference as to how DOS handles its own open files.

I'd say these never were DOS's handles in the first place. But that doesn't really matter. Implementing a DOS kernel to be remotely compatible to MS-DOS yields a lot of things you (the user) might find quirky, and I've to admit that often they are indeed quite quirky. The only way to manage that is good documentation of DOS's behaviour and its "internals" - i.e., these things that the user software might depend on. (I'd say it isn't much of an internal thing then anymore.)

Either you want to read that too, or I might get around to fix my code in the next days.

I'll look at what you have, but if you're going to be fixing the code anyway, send me a copy when you're done.

I'll go into the code this weekend a bit and send you what ever I'll have got then.

However, file processing still works even when re-using bytes 50h+ of the PSP.

Actually, that's only true if you move the DTA first. I ran into problems once when I tried to open a file while still processing the command line (testing a file name provided on the command line for validity) and it destroyed the command line. The default DTA is PSP:80h, the same place the command-line parameters are located.

I thought about file processing as "opening a file" (i.e. 21.6C, the only usable function) and all other services accessing a file, or anything which requires DOS to access the PSP for handle translation. You're right: When doing file finds or accessing files with some of the FCB functions the DTA needs to be set to a correct buffer.

Actually, after the memory has been moved around and the program terminated, does any part of the PSP really need to be preserved any more (even the stuff from 0h-50h)? The PSP might actually be a good place to put the TSR stack, especially if you could have the whole 256 (0-FFh) bytes instead of just 176 (50h-FFh). 176 might be enough, but 256 is definitely better -- best to not be skimpy with stack space.

No, in case you aren't "resurrecting" the process via Int21.50 later, you don't need the PSP. In your auto-relocation to UMA or my default relocation, the created memory block looks to MCB walkers like a process - there's no difference to a process's MCB. However, there doesn't need to be a PSP inside this block.

- 1 error because of TEST [87h],8 (masm syntax wants a DS: before the [87h])

He, yeah, NASM accepts that explicit segment specification too, but (last time I checked) will actually emit a (useless) segment override byte. Without ASSUME and implicit segments, it's always implicit that no segment specification means DS.

[Bret]

So it's not too difficult to make the source compatible with JWasm. The only true show stopper are the "duplicates".

The "duplicates" are what A86 calls local labels, which are used for short jumps, and can be duplicated (they're also the ones that use the ">" for forward short jumps). If JWASM doesn't allow local labels (which it sounds like it doesn't), maybe the "master" file could inject some random numbers or something to make them unique.

Write your suggestions now. I'd be interested in that VxD interface too.

The Windows VxD interface is discussed in RBIL under INT 20h, and the JLM interface provides a subset of these functions (described a little bit in the docs, but further details are in the source). JEMM currently only does this inside a JLM, for which japheth provides some library functions. I would like to see a real mode interface (could be just like the Windows INT 20h VxD interface, but wouldn't necessarily need to be be) that allowed access to these functions as well. I think the real-mode interface could simply be provided by a JLM (but don't know that for sure), rather than installing it natively in JEMM.

The replaced parts of the BIOS however do reside in Int09 (and possibly Int16), don't they? This means that programs which hook Int09 (not passing the request to underlying Int09 handlers) don't use the translated layout anyway.

The FreeDOS KEYB program has an option to intercept INT 09, but it only handles a couple of special keystrokes (like the Pause key). It passes most of the INT 09 functions to the original (BIOS) handler for processing.

Without ASSUME and implicit segments, it's always implicit that no segment specification means DS.

That's true for the CPU itself also, not just the assembler. ASSUMEs have always confused the heck out of me (maybe because I've never had to use them).

However, there doesn't need to be a PSP inside this block.

Sounds like the TSR stack is a really good way to use the PSP -- there is nothing to initialize. Unfortunately, it won't work for the USB host drivers, because they need a stack of at least 1024 bytes to call the PCI BIOS functions.

[japheth]

The "duplicates" are what A86 calls local labels, which are used for short jumps, and can be duplicated (they're also the ones that use the ">" for forward short jumps). If JWASM doesn't allow local labels (which it sounds like it doesn't), maybe the "master" file could inject some random numbers or something to make them unique.

Masm syntax knows local labels, but they exist inside PROCs only.

The Windows VxD interface is discussed in RBIL under INT 20h, and the JLM interface provides a subset of these functions (described a little bit in the docs, but further details are in the source). JEMM currently only does this inside a JLM, for which japheth provides some library functions. I would like to see a real mode interface (could be just like the Windows INT 20h VxD interface, but wouldn't necessarily need to be be) that allowed access to these functions as well. I think the real-mode interface could simply be provided by a JLM (but don't know that for sure), rather than installing it natively in JEMM.

I probably can provide a sample for a dynamically loadable JLM in Masm-syntax, which traps a port and notifies a real-mode TSR when an i/o occurs. If this helps ...

[Bret]

Masm syntax knows local labels, but they exist inside PROCs only.

In A86, a local label is simply defined by its format, since A86 doesn't use PROCs. A local label is a single letter followed by a number. I'm not real familiar with other assemblers, but I think some of them do similar formatting things, like preceding local labels with a period (".") or an at ("@"). I don't know that for certain, though.

I probably can provide a sample for a dynamically loadable JLM in Masm-syntax, which traps a port and notifies a real-mode TSR when an i/o occurs. If this helps ...

That would definitely help, though is not quite what the I/O Virtualization spec needs. The notification code is not called in real mode, it is called in 16-bit protected mode. It would actually be better to be called in real mode, I think, though it doesn't actually make much difference in how you write the handler. Here's a link to a discussion and some resources:

http://www.programmersheaven.com/mb/x86_asm/376018/376018/virtualize-io-in-dos/?S=B20000

The source for USBJSTIK has a working implementation. The spec could also use some improvement, since it lacks some features I think it should have (like being able to figure out which ports are being virtualized).

EDIT:

FWIW, I personally would rather see the complete VxD implementation from real mode before I saw the I/O Virtualization function. With the VxD, I'm not even sure the I/O function would be needed. I don't know of any applications that use the I/O function except USBJSTIK (I don't doubt that there are some, but I have never seen any). It's also easier to deal with a real mode interface than a 16-bit protected mode interface, since you can use segments instead of selectors which are much easier to understand and manipulate and manage in a 16-bit environment.

Also, FWIW, I would like to see some detailed documentation on the VxD functions. What's in RBIL is very incomplete in terms of what parameters/registers are used for input and output, details and hints and explanations, sample code (even psuedo-code), etc. I've done some googling, but haven't been able to find anything. You can glean some of the details by looking at your source code, but it would be nice to have a concise, detailed summary. I imagine you must have some form of detailed documentation, even if it's just notes from reverse engineering.

Also, I can understand if you don't really want to do this, because you would prefer everything to be a JLM. From my persepctive, though, that's a problem. I believe my programs need to work any any environment, including plain real mode DOS with no UMB's or extended memory available at all. I also don't want to write two programs -- one a JLM and the other not. I want to be able to take advantage of JEMM's extended functionality if it's available, but want to be able to work with limited capabilities even if it's not.

[c.m]

The replaced parts of the BIOS however do reside in Int09 (and possibly Int16), don't they? This means that programs which hook Int09 (not passing the request to underlying Int09 handlers) don't use the translated layout anyway.

The FreeDOS KEYB program has an option to intercept INT 09, but it only handles a couple of special keystrokes (like the Pause key). It passes most of the INT 09 functions to the original (BIOS) handler for processing.

That's interesting, but not related to my question. If an application directly accesses Int09, you can't translate the keyboard for it, right?

Without ASSUME and implicit segments, it's always implicit that no segment specification means DS.

That's true for the CPU itself also, not just the assembler.

... and these are the parts of an assembler I prefer "simple". Don't try to emit segment overrides automatically or "intelligently" - or at least let me switch of this feature.

However, there doesn't need to be a PSP inside this block.

Sounds like the TSR stack is a really good way to use the PSP -- there is nothing to initialize.

It might be useful to set the first two bytes of the PSP to something different from 20CDh, though. This value is used by some MCB walking software to determine if there really is a PSP, and since your stack will overwrite and corrupt the former PSP, we don't want this software to think there is one. Note that you can even change this value before terminating the process, since DOS doesn't (and doesn't need to) verify a PSP by this value - it simply assumes the passed PSP segment does contain a valid PSP at that time.

In A86, a local label is simply defined by its format, since A86 doesn't use PROCs. A local label is a single letter followed by a number. I'm not real familiar with other assemblers, but I think some of them do similar formatting things, like preceding local labels with a period (".") or an at ("@"). I don't know that for certain, though.

In NASM, there don't exist PROCs either. As you stated, a local label is defined by preceding a usual label with a period. As opposed (?) to A86 any local label must be unique inside the same non-local label. The local labels actually are expanded to labels of the form "label.local", where you specified ".local" as a local label after the label "label". This allows to reference local labels from any place in the source by specifying the full label, and also allows to specify a local label in front of it's associated non-local label (or after another non-local label). This feature is used by the source I provided above for a simple loop, where the actual (non-local) label is specified between the code, instead of in front of it.

[japheth]

I've uploaded the sample - it's called IOTRAP and can be found in the Jemm v5.73 source package.

That would definitely help, though is not quite what the I/O Virtualization spec needs. The notification code is not called in real mode, it is called in 16-bit protected mode. It would actually be better to be called in real mode, I think, though it doesn't actually make much difference in how you write the handler. Here's a link to a discussion and some resources:

EDIT:

FWIW, I personally would rather see the complete VxD implementation from real mode before I saw the I/O Virtualization function. With the VxD, I'm not even sure the I/O function would be needed. I don't know of any applications that use the I/O function except USBJSTIK (I don't doubt that there are some, but I have never seen any). It's also easier to deal with a real mode interface than a 16-bit protected mode interface, since you can use segments instead of selectors which are much easier to understand and manipulate and manage in a 16-bit environment.

Actually, I switched the sample so the DOS TSR is also called in 16-bit protected-mode. This is because calling in in "v86-mode" is somewhat nonsense because then port access would be still protected and an access would cause another trap.

Also, FWIW, I would like to see some detailed documentation on the VxD functions. What's in RBIL is very incomplete in terms of what parameters/registers are used for input and output, details and hints and explanations, sample code (even psuedo-code), etc. I've done some googling, but haven't been able to find anything. You can glean some of the details by looking at your source code, but it would be nice to have a concise, detailed summary. I imagine you must have some form of detailed documentation, even if it's just notes from reverse engineering.

Also, I can understand if you don't really want to do this, because you would prefer everything to be a JLM. From my persepctive, though, that's a problem. I believe my programs need to work any any environment, including plain real mode DOS with no UMB's or extended memory available at all. I also don't want to write two programs -- one a JLM and the other not. I want to be able to take advantage of JEMM's extended functionality if it's available, but want to be able to work with limited capabilities even if it's not.

To avoid having to write a documentation I just implemented - a subset of - the Win9X VxD API - so one can use the docs coming with the Win9X DDKs.

[Bret]

That's interesting, but not related to my question. If an application directly accesses Int09, you can't translate the keyboard for it, right?

It really depends on how it's implemented. If the application directly accesses INT 09, and doesn't call Int15.4F, and processes the entire keyboard itself (not just parts of it like FreeDOS KEYB does), then the keyboard can't be translated in that application using Int15.4F. If it does what FreeDOS KEYB does and only processes certain keys, then Int15.4F will work on the parts that the BIOS processes (actually FreeDOS KEYB does itself call Int15.4F, but not all applications do).

With my SCANCODE program, it actually is possible to translate keystrokes with any program no matter what it does, as long as the application doesn't intercept INT 09 in PM. SCANCODE can be set up to "wait" until after the application intercepts INT 09 before it intercepts it, so even though the application thinks it's completely controlling INT 09 it really isn't. That's not the main purpose of SCANCODE, but it can be used that way.

Don't try to emit segment overrides automatically or "intelligently"

Exactly -- that's why ASSUMEs always confused me. I never could understand why and when they would emit segment overrides, and if they were only going to do it when I wanted them to. It is perhaps a little bit "harder" to manage segments yourself, but it sure makes troubleshooting a lot easier.

The local labels actually are expanded to labels of the form "label.local", where you specified ".local" as a local label after the label "label".

That's not how A86 works. Local labels are really just a way of making sure JMP's and Jcc's are short. You can have duplicates even within the same non-local label. I'm not sure one implementation is inherently "better" than the other, just different.

**********************************

've uploaded the sample - it's called IOTRAP and can be found in the Jemm v5.73 source package.

I'll check it out.

To avoid having to write a documentation I just implemented - a subset of - the Win9X VxD API - so one can use the docs coming with the Win9X DDKs.

I've managed to find and download the WIndows 98SE DDK & Docs, but not a lot of apparently useful documentation. Any idea which files to look in?

[japheth]

I've managed to find and download the WIndows 98SE DDK & Docs, but not a lot of apparently useful documentation. Any idea which files to look in?

Open file Help\Other.chm, then navigate as the image shows below!

[Bret]

The install program didn't even create a help folder, but I finally managed to find it. Thanks.