PHP initially started out as a wrapper around Perl and gradually became a language in its own right. It has evolved to a point it can pretty much do anything you'd expect from a scripting language, even besides web development.
Note: I'm not arguing for or against the design principles of PHP, merely stating what I think I know. No flaming please
Perl started out as a text processing toolbox but has evolved into a full fledged programming language in its own right. People write all kinds of applications in Perl, including music players, Jabber clients etc. Note that this may also be achievable in PHP, you just would have to find libs for it. Most PHP libs are centered around web development.
As for your 2nd question, as girasquid said, you'd probably need a module like Tk for GUI applications in Perl. Even wxWidgets seems to do well in this regard. Perl being an interpreted language at its core, doesn't build an '.exe' file but the source pl file itself is executable. Note however, that there are tools which create some wrappers around the perl file to generate a Windows compatible exe file.