This is an old Win32 crackme, but still, has quite a lot of things that we can learn from it. I have solved it on IDA. You can download the target and the commented IDA database of this project. I will try to explain what you need to focus on if you are solving it without totally spoiling it for the people that, like me, are starting into the world of reverse engineering. After the explanation, I will link to the source code of the keygen, which is on Github.
I have noticed several usual patterns in this CrackMe that are interesting to know. First of all, this is a Win32 PE executable. That is still pretty common, though we are starting to see every day more and more x64 software out there.
The message loop
On Windows, everything is a window (yes, even the buttons and the inputs). They communicate with each other by sending messages that are processed in what is called the message loop. This message loops many times goes as you can see in the diagram. What can be seen on the graphic are Win32 API calls.
Understanding this is key in order to know how the program detects when we have pushed each of the menu items. As you can see on IDA, one of the parameters that are passed to the GetMessage function is a pointer to the place where the message is going to be stored. Such message contains several fields, but we are going to focus on two of them which are used to determine which element of the GUI is actuated on this crackme.
When creating the main window with CreateWindowEx, one of the parameters is the WndProc, or window procedure, which is the procedure which will manage the messages for the main window, by clicking on its address, we will see that there is one parameter called Msg, which points to the message that is going to be processed. Then we can see that there is a waterfall of jumps that check different things. Every time that there is a message dispatched to the window is processed by this procedure.
We can assume that what is there is an instance of an MSG Windows structure. First, the type of message is checked, and then the WParam, which gives additional information of a specific type of message. Is left as an exercise for the reader to know how the application knows when to show the “register” dialog and when we have clicked the OK button in such dialog.
String manipulation pattern
At the end, the name/serial check has to deal with strings. This is a very common pattern in many crackmes and thus deserves its own analysis. As you know, internally, strings are a succession of bytes (encoded using ASCII in this case, though it can also be Unicode) that ends with the null byte. This fact is exploited in order to go through such strings char by char. The loops usually look this way (there can be variations in the used registers or the registers that return values, but the pattern remains).
In the program, two checksums are calculated using this technique. The first, which is on the address 0x0040137E, calculates the checksum for the name. The pseudocode of the used algorithm goes as follows.
result = 0 foreach char in name: if char < 0x41: ERROR(Invalid Name) else: if char > 0x5A: char = char - 0x20 result = result + char result = result XOR 0x5678 return result
The second, in 0x004013D8, calculates the checksum for the serial with the following procedure:
result = 0 foreach char in serial: result = result * 0xA result = result + (char - 0x30) result = result XOR 0x1234 return result
What this last algorithm do is transforming the string into a number. First, multiplies by 10 in order to make space for the number and then adds the current number, transforming from its ASCII code to the number it represents. This is done by subtracting 0x30. This works because of the design of the ASCII table. 0x30 is the char “0”, 0x31 is “1” and so on.
If both checksums are equal then the serial is correct and shows the proper message. It is another exercise for the reader to translate the assembly code to this pseudocode. In the commented IDA database there is more than enough hints for that to be easy enough even for a beginner.
Reversing the check
Reversing this check is very easy, as the XOR operation is reversible. This means that if A XOR B = C then C XOR B = A. This is a nice property of this binary operator, and thus is commonly used. In order to create a keygen, it is enough to calculate the checksum of the name and then XOR it with 0x1234. This will give use the number for which the result of both algorithms will be equal. You can check the source code of the keygen (in C#) here.
I hope this series of articles are helpful, specially for the beginners. They will continue as soon as possible.