Vista Security

Posted in The Evil Empire on Fri, June 29, 2007 at 11:40 PM

One thing I forgot to mention in my previous Vista post was that I was able to debug the problem with my app in about 6 hours. Not surprisingly, the source of the issue was an incompatibility with previous versions of Windows due to security policy changes in Vista.

I may or may not have mentioned this in a prior post, but this particular app does some magic to determine what (if any) text is being drawn directly under the mouse cursor, in any window, from any app. It can then look up the definition for the particular word the user is interested in from a database. In order to do this, my app does some Win32 trickery to inject a DLL into every running GUI process. Once my code gets into every process, I then hook some Win32 API functions (in other words, whenever certain Win32 functions get called, my code gets called instead).

The tried and true program I usually test against is notepad.exe. Turns out that under Vista, my app and notepad.exe weren't getting along. At first it seemed that the API hooks weren't working, but attaching the debugger to notepad.exe quickly proved that false. I eventually determined why notepad.exe wasn't working, and really there's not much I can do about it right now. If I have some major time I can probably find a workaround, but it won't be pretty.

I next decided to try testing against Vista's version of Internet Explorer, which is probably the single most important program this application needs to work with (I personally don't use the mess that is IE, but unfortunately most people do). A bit of debugging showed me that IE did not suffer from the same problem as notepad.exe. For some reason when the code tried to communicate something back to the main app, it was never received on the other end. Because the API hooks actually run in other processes (in this case, in iexplore.exe), IPC must be used to communicate info back to the main application. I originally chose to use the WM_COPYDATA message, which is probably the simplest IPC mechanism you can use in Windows, which worked well enough until now. It was as if my application was never receiving WM_COPYDATA messages from iexplore.exe sent by my hook code.

After a bit of googling, I found others who had problems with SendMessage from one process to another in Vista. Most people reported that this was caused by differing security levels between applications. Under Vista, a "low integrity" process is not allowed to send messages to a process of higher integrity. This suggested that IE ran as a low privilege process in Vista, or at least with lower privileges than my application did. A bit more googling revealed that yes, under Vista, IE runs in "protected mode" as a low integrity process. I next turned off protected mode, but that still didn't solve the problem.

Luckily, I came upon a post describing the ChangeWindowMessageFilter API, available only under Vista. This allows you to instruct a higher priority process which messages it should allow to be received from lower integrity processes. Calling this from the main application with WM_COPYDATA (and a few other message types that I use) solved the problem.

While it makes complete sense to me why IE should run as a low privilege process (if IE gets compromised by some kind of rogue ActiveX control, for instance, it can start using window messages or other IPC methods to get access to information, perhaps sensitive, from other running processes), you have to wonder what the cost of such fine-grained security is. Windows is of course message-driven. If each message received by an application is checked for security, that's got to amount to a lot of overhead. More evidence as to why Vista is slower than XP? I think so.

Post a Comment