[notnot]

Hi, I'm trying to call some functions in an external COM library, but for some reason after the functions are called, the stack is screwed up and my program crashes.

The way the SDK library is set up, you first do a LoadLibrary(), then a GetProcAddress() on a particular function. You then pass that function a pointer to a vtable, which is filled with a bunch of pointers to functions within the DLL.

Well, the vtable is getting filled with the correct function pointers... but when I call the library functions, they both a) don't work properly and b ) mess up the stack, causing my program to crash.

I've never worked dealt with COM libraries before, so I don't know if there's something special I need to do. I tried modifying the function pointer definitions to be either __stdcall or __cdecl, but neither fix the problem.

I tried compiling the program both in gcc and VC++, it works on neither. VC++ gives me Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention even though I have verified that the calling convention is correct.

#include <stdio.h>
#include <windows.h>
#include "tgsdkx.h"

typedef HRESULT ( WINAPI *GetInterfaceProc )(LPSTR, LPSTR, LPVOID*);

struct textCapLib {

	 HINSTANCE lib;
	 ITextGRABSDK *sdk;
} textCapLib;

int main(int argc, char **argv) {

	 BSTR text = NULL;
	 HINSTANCE lib = textCapLib.lib = LoadLibrary("tgsdk.dll");
	 if (lib == NULL) {
	  return 1;
	 }

	 /* Now get the address of the GetInterface() function. */
	 GetInterfaceProc GetInterface;
	 if ((GetInterface = (GetInterfaceProc)GetProcAddress(lib, "GetInterface")) == NULL) {
	  goto ERROR_LIB;
	 }

	 /* Now call GetInterface(). It'll fill up our SDK structure with function pointers and stuff. */
	 if (FAILED(GetInterface("", "", (LPVOID *)&(textCapLib.sdk)))) {
	  goto ERROR_LIB;
	 }

	 /* Call the functions. The function pointers are set properly, as the proper functions
	  * are being called, but the functions appear to be looking for arguments in the wrong
	  * location, and the stack is messed up after calling them as well. */ 
	 (textCapLib.sdk)->lpVtbl->CaptureFromHWND((textCapLib.sdk), (INT_PTR)GetDesktopWindow(), &text);
   
	 return 0;

 ERROR_LIB:
	 FreeLibrary(lib);
	 textCapLib.lib = NULL;
	 return 1;
}

Here is the tgsdk.h file, which was provided by the library developer for use. See especially the code starting at line 810 for the vtable definition.

One last thing: I ran my program through a debugger; it appears that the library functions are looking for arguments in the wrong position on the stack.

Any suggestions would be very much appreciated.

[bsaunders]

Doesn't CaptureFromHWND only take two arguments?

[notnot]

When calling from C++, yes. But the way the header file is set up, to call the function in C you've got to access the vtable, and pass a pointer to the ITextGRABSDK instance (this) as well... so the way I'm calling it is -- as far as I know -- correct.

So, in C++, you'd just do:

(textCapLib.sdk)->CaptureFromHWND(arg1, arg2);

But in C you have to do:

(textCapLib.sdk)->lpVtbl->CaptureFromHWND((textCapLib.sdk), arg1, arg2);

[skaoth]

What you are doing doesn't look like a COM call to me. This simply looks like calling a DLL function directly.
Ususally when you call a COM object you use CoCreateInstance() to get a reference to the COM object you want.
You can take a look at http://www.netez.com...FAQ/bg_com.html to see a simple example of how
this is done.

Do you know what the Interface pointer is? I assume its ITextGRABSDK. Do you have the CLASS id?
You can see how to create the COM object here http://msdn.microsof...615(VS.85).aspx

[mikeblas]

Right; it looks like you're using a DLL, not a COM object. Why aren't you just implicitly linking to the import library, and calling the members that way?

[notnot]

Well... I believe this is the way I'm supposed to be calling the library, as they provide an example that uses this method. For some reason, my version screws up the stack, while theirs apparently doesn't.

[perfectly.insane]

I don't see anything that is obviously wrong here. While one normally uses CoCreateInstance/CoGetClassObject when working with COM, there are ways around such (such as having other functions do that work for you, or possibly even performing the work of these two functions yourself).

COM class methods should be of type __stdcall. In MSVC, this means two things. Other than the well known fact that __stdcall functions clean up their own stack (and that they tend to have decorated names indicating the memory displacement of the arguments to be passed), in MSVC, the this pointer is passed on the stack. Normally for MSVC, on x86 systems, it is passed in the ECX register. The header generated by MIDL appears to be correct. I would hope that their header file for this matches the MIDL generated one in terms of calling convention (as I assume that this is a well tested piece of code (the COM library)).

I believe GCC always passes this on the stack for x86.

This post has been edited by perfectly.insane: 02 June 2009 - 07:06 PM

[mikeblas]

perfectly.insane, on 2 Jun, 2009 - 06:01 PM, said:

The header generated by MIDL appears to be correct.

I must have missed it; where was the MIDL-generated header posted?

[perfectly.insane]

mikeblas, on 3 Jun, 2009 - 12:50 AM, said:

perfectly.insane, on 2 Jun, 2009 - 06:01 PM, said:

The header generated by MIDL appears to be correct.

I must have missed it; where was the MIDL-generated header posted?

Check the link to the sdk header file. The generated by MIDL comment remains intact.