Defending software from piracy and abuse is a game of attrition. After all, unless you stick to remote server code you put everything the cracker needs right into his hands. Even when you don't provide public downloads you can't trust your legitimate users, who may well employ less scrupulous people to do their burger flipping. So you want to take advantage of as much security as you can afford, and you do this in layers that impede various points of access.
Remember, people who insist that "security by obscurity" isn't security all are probably trying to sell you some expensive and awkward cryptographic tool. Then you end up with passwords and keys for that which you have to secure through obscurity anyway!
The demise of Windows XP makes many new security technologies introduced with Vista practical to use in more applications, and Vista even improved on a few that came along during XP's long gasping at life. You can finally feel free to abort execution whenever your sensitive applications are being run on an insecure pre-Vista system!
Hijacking Stealth
One of the simple things you can do is add some stealth to thwart application hijacking.
We see forum thread after thread on the use of "spy" tools and window spelunking (window enumeration and pattern sniffing) to try to get the handles of controls in your application to try to make it dance like a puppet. The usual motivation is try to steal functionality from the free EXE version of software you have written when they want to avoid paying for the licensed full-function EXE with an Automation interface or DLL you sell. Don't discount the copy/paste coder who simply can't handle an API and finds copy/paste hijacks easier to use instead either.
VB6 comes with a set of "windowless controls" that were primarily meant for use in custom OCXs meant to be run within the context of IE web pages. These duplicate a number of the intrinsic VB6 controls, and while Microsoft claims not to support them they still work just fine even as late as Windows 7 and probably Windows 8.1. They don't replace all other controls and they have a few quirks here and there, but use as many of them as practical - they're mostly "drop in replacements" for the intrinsic controls.
Code Stealth
Another common defensive technique is to try to make code spelunking harder.
This usually consists of combining some manual obfuscation with native code compiles. Sensitive logic can be housed in classes with obscure and misleading names. Every little bit helps, and to keep the burden on yourself lower you might restrict this practice to your most sensitive logic, such as anything dealing with encryption keys. Class names that look like a hex value (e.g. "D009A67C") can be effective, or even proper names can work ("Cynthia" or "AbdulHakkim" or "BabeRuth"). Silly? Not as much as you might think.
Data Stealth
And what about data?
Well, some of it may be sensitive and the rest of it not so much. Some may be within your EXE and the rest in external files. Focus on the most sensitive information to get the most bang for your extra effort.
Just like class names, you can obscure data (particularly text) in your EXE to help fend off those code spelunkers. You can put data into custom resources in compressed and/or obfuscated form. Even XORing text with a decently long "key" pattern is better than doing nothing. When you only need ASCII text then ANSI is safe, though if you actually use characters above &H7F you might be better off using UTF-8 encoding before XORing.
The same can apply to data in external files, but you might want to use somewhat stronger encryption. Don't go overboard though. There is no reason to make things too hard on yourself unless you're guarding State Secrets.
How Can Windows Help?
Beginning prior to Vista but maturing since then we have a number of stealthing and hardening techniques available. Most of these defend against attempts to probe or modify your code to take control of it through modification or hacks like DLL Injection.
Code Signing
This isn't always practical because you need to acquire and maintain code signing certificates as well as guard your "reputation" to avoid certificate expiration or revokation. Still, it is the only tool Windows offers to guard your EXE against modification by a cracker.
However even this isn't as valuable as it might be. The "bar to entry" is so high that users are accustomed to installing and running programs that are not signed at all or have suspicious (self-signed, expired, etc.) certificates.
Hardware Protections
This is really the NX ("no execute") or DEP ("Data Execution Prevention") technology that was added to x86 and x64 CPUs over a decade ago. It isn't compatible with all programs, but any clean VB6 program and most of the libraries it would use certainly should be.
Starting with Vista, Windows has supplemented this with ASLR (Address Space Layout Randomization) or Dynamic Basing. This makes it tougher to probe your code using cracker tools that use techniques originally intended for binary debuggers.
Both DEP and ASLR are opt-in technologies, though the savvy PC user has changed his system DEP setting from Windows Client "opt-in" to "opt-out" so that DEP is applied to processes by default.
Since you can't trust users (after all the crackers are "users") you can do a few things.
When your EXE is linked you can specify "Hard DEP" and opt into ASLR. When your EXE is run you can detect whether DEP is active and if not try to turn on "Soft DEP" and if this fails terminate excution.
More Windows Protections
Structured Exception Handler Overwrite Protection (SEHOP) must be done by your application installer. It involves writing a registry key into HKLM.
Heap Metadata Protection can be accomplished in code when your program starts.
At this point I suggest you read the Microsoft white paper that was produced a few years ago after Vista technology was finally widely deployed:
Windows ISV Software Security Defenses
This brings together many of the new application security features Vista brought to the table. I'm a little shocked that these aren't very well known to VB6 programmers, I haven't found anything on the topic after many web searches. Though aimed at C/C++ programmers much of it also applies to other native code compilers.
The Code
Finally we've reached the sample code. Here are several items you can use to help implement some of Microsoft's recommendations that apply to VB6 programs.
Using the HullIntegrity class
Create an instance, call the RaiseShields() method.
RaiseShields() returns True when heap corruption detection and DEP have successfully been opted into or were already in effect.
If it returns False then FailReason and LastSysError can be used to perform additional failure analysis programmatically, or for reporting to the user why your program has decided not to run.
Corbomite = True means "permanent DEP" was set on PE file.
Maximum = True means running under Windows Vista or later. Might not return True when your application runs with appcompat version lies (XP, etc.).
The VerifyHullIntegrity demo Project illustrates basic use.
Ideally you would terminate if RaiseShields() returns False or if Corbomite or Maximum are False after the call. Terminating when run under Windows XP is a judgement call, but a safer one each day now that XP support has ended. You might choose to set Maximum = True even for Windows XP SP3 by calling GetVersionEx() and evaluating the results of that. XP SP2 and earlier aren't nearly as safe though.
Using the HardenPE utility
This can be as simple as dragging your compiled EXE's, DLL's, or OCX's icon onto HardenPE.exe's icon and dropping it there. Running HardenPE.exe with no dropped file will let you browse to the PE file to be "hardened."
You could easily make a non-GUI version of this for use in scripted builds.
Using the QueryPE utility
Just like using hardenPE but it doesn't make any changes. It just reports the current status of the DEP and ASLR attributes.
DEP opt-in, Address space layout randomization opt-in
Can be accomplished at runtime via HullIntegrity.RaiseShields() call.
Better accomplished by "hard opt-in" which HardenPE.exe does for you.
/SAFESEH exception handler protection
Not tested with VB6 EXEs. Requires a newer Link.exe than VB6 ships with (version 8.0 or later?) and interception of the Link run during VB6 *Make* to use this switch.
SEHOP opt-in
Not tested. Accomplished during installation by writing a special registry entry for your application.
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\
Image File Execution Options\
MyExecutable.exe]
"DisableExceptionChainValidation"=dword:00000000
Heap corruption detection
Can be accomplished at runtime via HullIntegrity.RaiseShields() call.
Testing
- Compile all three Projects: HardenPE, QueryPE, and VerifyHullIntegrity.
- Run VerifyHullIntegrity and examine the MsgBox.
- Drag VerifyHullIntegrity.exe onto Query.PE.exe, drop. Examine the MsgBox.
- Drag VerifyHullIntegrity.exe into HardenPE.exe, drop. Examine the MsgBox.
- Run VerifyHullIntegrity again, examine the MsgBox.
- Drag VerifyHullIntegrity.exe onto Query.PE.exe, drop again. Examine the MsgBox.
Additional Reading
On the effectiveness of DEP and ASLR
Quote:
DEP (Data Execution Prevention) and ASLR (Address Space Layout Randomization) have proven themselves to be important and effective countermeasures against the types of exploits that we see in the wild today. Of course, any useful mitigation technology will attract scrutiny, and over the past year there has been an increasing amount of research and discussion on the subject of bypassing DEP and ASLR. In this blog post we wanted to spend some time discussing the effectiveness of these mitigations by providing some context for the bypass techniques that have been outlined in attack research.
Inside the Windows Vista Kernel: Part 3
Quote:
This series has so far covered Windows Vista kernel enhancements related to processes, I/O, memory management, system startup, shutdown, and power management. In this third and final installment, I take a look at features and improvements in the areas of reliability, recovery, and security.