Cross compile mingw

This is mostly a page of notes on what I did. Once it's working I'll replace this page with something more useful.

I don't really, really, really hate Windows. However, I'll do just about anything not to use it not even with Mingw and friends or Cygwin. And the Windows I do have is old XP (SP2) in a virtual machine (slower) without virus protection so I don't like to leave it running for any length of time. Did I mention the damn nagware from HP and Oracle about my print drivers being out of date or a new version of Java is available and so on. Then when you go to shutdown it may install patches from the mothership which can take forever to finish. It just squeezes the fun out of using a computer. But, I don't hate it.

When I discovered that MingW can be used to cross compile windows source code on Linux I wanted to see if it could build Shoes without all that ugly stuff in the paragraph before this one. Evidently real live big time open source projects like VideoLan (VLC) and WxWidgets do this, so it can be done. Fortunately getting the mingW cross compile tools is just and apt-get away for debian and ubuntu users. Be sure to look at some of the links at that WxWidgets page about bewares and things you might need to think about. Turns out even Synaptic knows about those tools on my Ubuntu 11.04 system.

The next step is what to do about the Rakefile. It knows nothing about cross compiling. So the first task is to make a copy of shoes. If the experiment doesn't work. Throw it away. But one has to think about what if it does work? Well then, you want to be able to merge the code into the Shoes mainline without a lot of hassle. After thinking about it I decided I'll copy the existing rake file and call it xmingrake so that means if I want to build it would be $ rake -f xmingwrake and rake -f xmingwrake package and rake -f xmingwrake stub Or maybe I'll name it crossrake or crossmingw.

Since that rakefile will never build an OSX binary I can delete a lot of the mysterious OSX stuff. I'm feeling better already. Then I'll dup the existing make/mingw into make/crossmingw and delete some windows stuff that a cross compile doesn't need and it will all be good unless it all goes horribly wrong.

Which it did before I even got around to modifying the rakefile! Don't you hate it when that happens?
I'll need the Ruby 1.9.2 source code built with the cross compiler so I'll have proper ruby dlls to sling about. Now it's gotten complicated. The easy thing would be install the ruby-installer on my Windows VM and copy the Ruby dll's over to to Linux and use them for the link phase of shoes. If that hack works then it might be worthwhile to cross compile ruby in mingw, maybe. I just want to fix one error in a windows stub program that doesn't even use Ruby.

Already I'm gathering baggage and haven't done anything.

What I really did.

When you pick mingw32 in Synaptic (check and apply) it installs the mingw stuff for compiling both 32 bit and 64 bit. Even though I am running X64 I don't want to compile for 64 bit. So the CC to use in the rakefile(s) is /usr/bin/i586-mingw32msvc-cc

ccoupe@twb:~/Projects/shoes3.1/mingx$ /usr/bin/i586-mingw32msvc-cc --version i586-mingw32msvc-cc (GCC) 4.4.4

Hopefully that gcc is current enough. You might note that in ~/Projects/shoes3.1/ are two directories, shoes and mingx. The first one, shoes/ is where I build shoes (3.1) for linux by linux. Probably x64 by X64. The mingx/ directory was a cp -r of shoes/ In mingw/ is where all the damage will be inflicted. I created a mingw/make/crossmingw directory and cp -r from make/mingw to there. It's just a clone of Shoes except I mv Rakefile crossmingw and I tested that I can't just run rake in there. I need rake -f crossmingw. That reduces my chance of getting confused and raking without looking. I also did a ccoupe@twb:~/Projects/shoes3.1/mingx/make/crossmingw$ cp dlls dlls.org It's a list of all the dll's needed to be built by the cross compiled Shoes. Ick and I haven't gotten to the ruby install/compile mess or the copy from windows hack. I think I'll work on the stub task first because 1) that needs fixing and 2) a damn sight easier.

After thinking more, to be a viable Shoes development for Mingw I will need to build all those dll's or package dependencies (pango, cairo ...) with the cross compiler and also cross compile ruby. That's a whole lot of work. Fortunately, someone has done this before, Nokogiri developers and they describe it. It needs updating of course to current ruby versions and evidently it's doable in OSX as well as Linux.

It also means a butt load of source code to be maintained (copied from projects, cross compiled, debugged). Pretty much re-inventing what the ruby-installer project does. Of course a Shoes maintainer on Windows also has to deal with keeping the dependent libraries current. That's probably easier in Linux or OSX but still it's a lot of work. I'm not sure how to fit all that into the Shoes project. Should I make a new github project Shoes-Dependencies? (one could also look and see if its already been done, ala Nokogiri).

I'm seriously leaning to future users of the cross compile system have an rvm installed source code tree for the version of ruby being built. Reduces the download requirement. Another thing to think about.

Switching gears. Lets find the DLLs that should be cross built and copy them from Windows to a safe place on the linux box. I fired up the Windows VM and did a file search of Shoes and found some recent ones at C:\Program Files\Common Files\Shoes\0.r1514. I'll ftp them to the Linux at ~/Projects/shoes/mingdll (yes, ftp is faster than cranking up shares and virtual shared volumes). Unless you ftp wrong. Sigh, I really wish I was smart. And just to be mean, Windows is installing some damn update at shut down.

What are the .dlls ? Any Ubuntu info are from my 11.04 system. There may be newer version of the libraries. This is mostly documentation of what Shoes/MingW needs

• freetype6.dll Website
• libcairo-2.dll Website Ubuntu: libcairo-gobject2
• libeay32.dll Not on Ubuntu Part of OpenSSL
• libexpat-1.dll Website Ubuntu: libexpat1-dev XML parser
• libfontconfig-1.dll Website Ubuntu: libfontconfig1-dev
• libgio-2.0-0.dll Part of glib2
• libglib-2.0-0.dll Website
• libgmodule-2.0-0.dll Part of glib
• libgobject-2.0-0.dll Should be part of Cairo. Trust but verify.
• libgthread-2.0-0.dll Part of glib.
• libiconv2.dll no package in Ubuntu. It's a Windows thing. Source is available, but version?
• libjpeg-8.dll Wiki Says Source available everywhere
• libpango-1.0-0.dll Website Ubuntu libpango1.0-dev
• libpangocairo-1.0-0.dll Website
• libpangoft2-1.0-0.dll No Ubuntu equivalent. Not sure where this comes from. Might be part of Pango
• libpangowin32-1.0-0.dll No Ubuntu equivalent. Part of Pango or pangocairo?
• libpng14-14.dll Start here Ubuntu libpng12-dev. Version number demons appear!
• libportaudio-2.dll Website Ubuntu: libportaudio2 and libportaudio1.9-dev. Comforting that version number BS.
• libshoes.dll This is built by Shoes
• libssl32.dll Part of OpenSSL? PITA.
• libungif4.dll GNUWin32 Ubuntu: libgif-dev.
• msvcrt-ruby191.dll - This should be built from source for 1.9.3. Sigh.
• readline5.dll Start here Ubuntu: I have both readline5 and readline6 installed, but none of -dev packages. It's needed to compile ruby, even if it's not used by shoes.
• sqlite3.dll Website Ubuntu: 3.7.4-2. This should be built in Shoes/ext but I won't bet on that.
• zlib1.dll [Website](http://zlib.net/ Ubuntu: zlib1g-dev.
• zlib.dll What Happened With Zlib) Is is really needed?

Some of these can be gotten from http://gnuwin32.sourceforge.net/packages.html and it might be the better place. Or not. There is an extra directory or two compared with to the linux src tar ball that appears to be CYGWIN running on Windows stuff which might have clues but could be waste of time.

Alternately, you could use the GTK+ instructions

Not every fricking one of those has to be cross compiled, yet! Unless you want the latest and greatest. I will need the .h files to compile shoes against and the .h had better match what is in the .dll and the best way to that is to build the damn things and make sure the cross compiler finds those .h files and .dll before it finds my Ubuntu .h files which might be newer or older. That implies that I have the .h in one directory for the cross compiler and that means I need to download the source and copy the .h into that cross compiler includes directory. Might as well compile all the dependencies -- It's going to cascade to cross compiling them all. Suck it up, Recruit!

I'm guessing they all use the autoconf stuff. You know, the ./configure; make; sudo make install. SUDO make install is absolutely a good way to litter a nicely running Linux with windows crappola. When dealing with cross compiled libraries, Make install should copy the .h and .so or .dll to the happy place where only them Mingw cross compiler will find them. For me I think ~/Projects/xmingw/[include|lib] is the place. in the same directory will be the ruby1.9.3/, pangox.y.z/. and any of the other source code projects needed by the Shoes crossrake. I'll need a test case to get the autoconf and cross compiler settings dialed in. Perhaps the zlib madness before I tackle cross compiling Ruby 1.9.3. Smart people would create a new Ubuntu account to do this in so that is no chance of a sudo FU. I'm probably not that smart. Time to compile something.

That involves dealing with autoconf (./configure --options). There are many web pages to read but I found theseGlib cross compiling. Since I might have to build Glib and the instructions aren't that difficult, that will be the first test. No. Something doesn't look right. More web reading... It seems what I really want is this replace his 'mips-elf' with appropriate values for mingw32. Seriously important is that clue about ac_tool_prefix=mips-elf-. Do not ignore that trailing '-'

As is expected, this is a can of worms with a side of order whup-ass. Time to revisit the plan with new information. Surface from the deep dive.

1. I want to cross compile Shoes with mingw (mostly the stubs) and get NSIS working so I can package a semi-new Shoes installer for Windows. The Shoes rake files can't handle that cross compile so I've set up my own rakefile structure.
2. I don't have to rebuild all the DLL's or even any at this point I just have to link against them.
3. Eventually, I want to build Ruby 1.9.3 and use that to compile and link shoes. At that time, I'll have to deal with autoconf/configure and Ruby's mkconfig.rb.
4. I may have to build all the other dependencies. But not until I've done steps 1,2, and 3.

There are clues I picked up on the shallow dive into autoconf so that it is not wasted time. As much as I want to do everything at once, you have to start at step 1. Sadly autoconf and configure makes more sense than the Shoes rakefile.

January 15, 2012

I do not like the Rakefile re factoring in Shoes 3.1. It might make sense to someone but not to me. Even less sense for a cross compile. There are no tests that will execute in that environment. Having to install the bundler gem and run it (and discover you need to apt-get libxslt-dev) is just a waste of effort. I'm sorely tempted to make a new rake file that doesn't include all the horse shit. On the other hand I'm that close to getting it to work. Real soon now...

Yes! I've got something that doesn't fail on rake -n -f crossming That does not mean it works. It only means that rake isn't complaining. Now I can set CC, Include and lib to point to where it can really fail. And it will big time, since I don't have all the .h files needed in my includes directory that match the dlls in lib that cross compiled Shoes wants to link with. I should also mention there is no :stub task in the rakefile. YET.

I need to get make/xmingw/env.rb sorted out first. I can almost feel the hacking spirit waiting to grind me down, one .h file at a time. I expect that. I have a plan for that (subject to reality checks.

Yes! It starts the cross compile and stumbles at the lack of a cairo.h (set your terminal pref to unlimited buffer). This is of course because I don't have cairo.h and now the devil meets the road. I could copy a cairo.h from my running linux system or I could download and cross compile Cairo source which mostly depends on all all the other things being cross compiled. I might get away with using the linux cairo.h to compile and in an alternate universe filled with hopeium, it'll just work. Of course it won't for long, hopeium never does. On the third hand I got cross compiler error messages and my make/xmigw/env.rb hacks look more like insightful modifications and my rake file hacking hasn't been hacking but a well controlled expansion of Shoes build capabilities. Not all the mods have been made to my xmigw env.rb or tasks.rb but it is clear that cross compiling shoes/mingw really could be an achievable goal.

January, 16, 2012

Time for me to download and cross compile all the dependencies and learn to love autoconf with a cross compiler and then I can get back to fixing the rakefiles in my xmingw sandbox. Or I can download and try the GTK dev packages. I think I'll try the GTK+ versions since I vaguely remember that's where the Shoes Windows developer ashbb got his. I downloaded the giant bundle offered. I got luck with unzip. I keep forgetting that zips rarely have a single parent directory like proper tarballs do. You need to be in the directory first, in my case that's /home/ccoupe/projects/xmingw. FWIW the dll's seem to be in /bin instead of lib/

The first fix of my fixes is to not use the '~/' shortcut, use the absolute path. I also had to set ENV['SHOES_DEPS_PATH']='/home/ccoupe/projects/xmingw' in the proper place in my the top level rakefile crossmingw. Have I mentioned I don't like the new rakefiles? Now I'm failing because my system ruby (rvm installed1.9.1) doesn't have a ruby/win32.h called from ruby/defines.h. If that file is the result of the ruby config process I'm not surprised. It was always going to get down to cross building a Ruby that was Windows ready. Actually, I'm pleased with the progress I've made.

Since I've just willy nilly grabbed the latest GTK+ versions of includes and dlls, there really isn't much hope for any kind of compatibility with the current version of Shoes Windows. I might as well work on cross compiling Ruby 1.9.3. Oh I'll have to modify the rakefiles to use the MingW32 compiled Ruby and hope that extconf.rb in the Shoes/ext behaves but that's a worry from the future.

So I downloaded Ruby 1.9.3 source code and I'll try this recipe after adjusting for Linux instead of OSX and after reading the Install and Read me. Doesn't sound like I can avoid tinkering with Shoes' built in gems.

I'm having some problems with the Ruby build in the recipes for cross compiling. In ubuntu I chose to install autoconf and autotools-dev. Those are blunt tools to be only used when you know what you're doing. It's best not go in those weeds half cocked. Been there before and this time I'm going to not go there. The ./configure almost works and then it links miniruby.exe and tries to run it. It also appears to use ruby 1.9.1 to do that. I will have to dig to where my configure has gone wrong.

January 17, 2012

I managed to cross compile Ruby 1.9.3 !! Mostly. For some reason or bug it wanted to install binaries into /usr/local/bin. Good thing I wasn't su. It also failed to build some ext/ that I think Shoes should have. It failed to build curses, dbm, ffi, fiddle, gdbm, iconv, openssl, yaml, psych, pty, readline, syslog, tk and friends and zlib. Curses,pty, and tk don't concern me. fiddle and psych? ffi is probably not Shoes No idea what they do. The rest of them bother me. Some of those have dll's in the Windows 1.9.1 Shoes suggesting that yes, shoes does need them, or used to.

I need to document what works this far. In the ruby-1.9.3-p0 directory in ~/Projects/xmingw I created a script crosscompile.sh with #!/bin/sh env ac_cv_func_getpgrp_void=no \ ac_cv_func_setpgrp_void=yes \ rb_cv_negative_time_t=no \ ac_cv_func_memcmp_working=yes \ rb_cv_binary_elf=no \ ./configure \ --host=i586-mingw32msvc \ --build=i686-linux \ --prefix=/home/ccoupe/Projects/xmingw make ruby make rubyw.exe make install

GRRR. I can't get a code block working in Markdown. That's 30 minutes lost. GRRR

A couple of points to note. --build and --host must be different to tell configure that it's cross compiling. --build is pretty much ignored (it guess what you are compile on) but something has to be specified. For me, the --host string is the prefix to finding where the cross compile tools live. Once configure finds them then things get build as i386-mingw32. Go figure. You'll also see some ruby libs/dlls are named with 1.9.1 instead of 1.9.3. `linking shared-library msvcrt-ruby191.dll' for example. That also happens on a Linux build for Linux. A well known sigh.

Still, it's solid progress with two problems to solve.

1. Make install
2. The extensions not built.

Turns out on my rvm installed Linux pysch isn't installed either and require 'yaml' complains in irb about pysch missing. That's a serious fail for 1.9.3-p0 (rvm). 1.9.2 has no problem with require 'yaml'. Turns out the might be a contentious issue of the ruby intelligentsia. You know the folks who that think the bleeding edge will solve problems and it's not really a bleeding edge knife gushing wound. "It's only a scratch," they say. "We'll fix it in the next patch release." I know the pattern of wishful thinking.

I also know the way forward is to build on things that work now! For me, I'm dropping back to Ruby 1.9.2 and see what cross compile errors that presents. Pysch might still be a problem even there for cross compiling. Suggestions are 'bundler' has a role to play in the mess. I removed the 1.9.3 source code directory and asked rvm to remove it.

Yes, yes! Cross compiling 1.9.2 is so much more informative and it gets the pysch and slyck right and and maybe opensll and all other others. If you are using rvm, make sure it matches the ruby you are trying cross compile. Seems obvious but even an old ruby will work until it gets to building Rdoc/RI/. That said it still died trying to install into /usr/local/. On the Shining Hand it is seriously faster to build 1.9.2 compared to 1.9.3. I noticed that 1.9.3 build speed (lack of) when installing 1.9.3 with rvm so it's not my imagination or my cross compiling horseshit. 1.9.3p0 building is broken. Broken in rvm and broken for cross compile and seriously slow.

About the broken cross compile in 1.9.2 (.3?). Did you know you have to apt-get install libyaml-dev and bison to cross compile ruby? Bison?

January 18, 2012

I fixed the ruby install bug. Of course it was my typo. I still don't know if my Ruby 1.9.2 is right for Shoes but it's time to build shoes and fix the rakefiles and then fix stub.c/blank.exe but first the rake files. I'm not going to document each fix here unless its a non obvious.

Now we have real problem to solve! #include <winhttp.h> MingW32 doesn't have it. The winhttp.c code is only used in the stub but that winhttp.h file is included from all kinds of nested includes. I see what ashbb was dealing with when he made Shoes Policeman for Windows. For the C unfamiliar the angle bracket includes only looks in the compilers defined directories. In this case, MingW system includes. In Microsoft compilers it will be where ever MSFT decides what is in the system includes. Since its Shoes code that includes this, the shoes code can be changed from angle brackets to quotes and change the name to avoid conflicts with shoes "winhttp.h". It's confusing but the angle bracket include could be change to a Shoes relative quote brackets include.

I suspect the windows api glue does not exist in Mingw and there might be some licensing issues. I'll bet winhttp.c and h can be recoded in MingW friendly ways. Http over tcp has been done a lot of times and something open source friendly might fit in. After changing the Shoes includes and the rakefile when compiled by mingw. I believe ashbb did this when he reinvented his new stubs. But first, check the interwebs to see if it's been done before.

January 19, 2012

I believe I have a solution for those Windows include files: Wine has them and I've got Wine installed. They are LGPL and any structs and constants would have to consisted with Windows otherwise Wine wouldn't work. It's worth trying. That means I'll uses the old Raisins stubs, updated of course to the the proper URL for Policeman. If that fails, it may be possible to build an stripped down curl library and use that.

Some compiler success. I copied my linux .h files for gif_lib and jpeg (multiples). It got far enough to try linking libshoes.dll. Of course it died because libungif and libjepg aren't built. Cross compiling libungif was a bit confusing. I used my crosscompile.sh script to run the configure and make and make install. That died when it tried to build some .exe ? Turns out that project has subprojects and generated Makefiles. I cd into lib and did a make and make install and it built and installed the library which the Shoes build didn't choke on. I suspect there could be more work will be required with libungif. Libjpeg wasn't that difficult using my crosscompile.sh script.

Have I mentioned that I hate the Policeman re-factored rake files? I don't think they been tested with Windows. Much less a cross compiler. It gets worse for me because I don't understand linking in Windows and to fry my ass, winhttp is among the missing. Not really a surprise. I expected it because using the wine headers only gets the compile to work. I did not expect all the other link errors and the rakefile opacity. Let the cool kids sort it out if they can or care too. I'm calling it a fail because I fail to care enough about Windows.

My backup plan to replace winthttp (the part that Shoes needs) with a stripped down curl static lib still has merit but it would need to be done on a Window/MingW system to make testing easy and I'm just not up for that degree of pain.

I've packaged up my rakefile, 'crossmingw' and it's dependent rake crud along with the crosscompile,sh script to invoke when you cross compile the dependent projects. At my website It's a teeny tiny archive that expands into ./crossattempt/ you are expected to understand where to move the files. They are only hints of what might work, except they don't.

I grow weary about Shows/Windows. I gave it a shot. I'm going Walkabout on Shoes. Might be back, might not.

Jan 27, 2012

I hate quitting. You know, when you break up with a girl friend because you can't see a positive long term outcome by staying together? So just move on and then you call her a week later to make up and even then you don't believe it's really going to last? Perhaps that hasn't happened to you. We'll thats me and Red Shoes and MingW.

After a week of pretending I don't miss Shoes, I managed to get the link errors reduced to not having the winhttp.dll to link against (plus some warnings about the Ruby dll). It's not like I haven't mentioned that old boyfriend Winhttp before. Winhttp is used in two places. The Shoes download command (for images or whatever) in in a Shoes script) and in the stub that downloads Shoes.exe on a net install.

Assume I had a curl.dll for Windows. For the Shoes download command, I'd just use the native/linux code (or a copy of it) to compile and link against. The curl.dll compiled by Mingw does exist elsewhere so I can duplicate that. I Think. That would involve changes to the Shoes source code and not just my rake files. It would require Windows developers if there any, to compile curl with their MSYS/MingGW system. Assuming we can cross compile curl for Shoes purposes. I think we can.

That leaves the stub.c or stub-inject.c, the thing that runs at net install time, that would need to have a curl library (static) inside when something is packaged. It would add to the .exe download size but that is probably not a big deal.

It's time to _get right_with github. I'm not sure of the best way to do that.

And then you run into How to cross compile

Jan 28, 2012

It turns out that replacing winhttp with Curl is not the easiest thing to do. It needs to be done eventually but just not right now. There are hints that I can create a winhttp.def and .a or something that is good enough to compile and link against and if done properly, will execute properly when run on a real Windows system. That involves wine and pexports.exe and dlltool and some hand editing of the .def. I almost understanding how that should work. FYI, the ruby 1.9.2 build I did probably needs some dlltool love too but those are only warnings from 'ld'

Hints here

Now I know more than I wanted to about dll's and Windows. It's kind of fun in a sad, sick way.

Jan 29, 2012

Fortunately the linker reports the winhttp.dll entry points so I just put them in the winhttp.def in place of the existing entries from pexports.exe. Of course in the stub.c calls an entry that I haven't mocked up then I'll have to do this over. It's not hard to do and for linking purposes, it works! The libshoes.dll builds. Whee!

There might be legal issues for distributing the winhttp.a built from a purloined winhttp.dll. Or not. Based on file sizes, the .a is just entry points to the .dll and not the code within the dll. As long as it works, I'm going to ignore the issue.

Now, it's time to fart around with the tasks.rb. make_resource() needed some simple help to call the cross compiler' windres compiler. Simples. make_app() is where it gets interesting. make_so() will be fun and so will make_install (). Not to mention the missing make_stub. In make_app() are things things I don't understand. Oh, I understand them for Linux but not so much for Windows and and the cross compiling will introduce confusion. I suspect there is a lot of useless crap happening for Windows or cross compiles to Windows in that rake file function/method.

Tracking down my latest build failure gets me to make/make.rb common_build(). WTF? Jesus Christ so loved me that he gave me these rakefiles!?! This is the point where the Shoes gems/extensions get built using the make_so() in the various rakefiles. In Java it would be an abstract class that bites your nuts off if you don't know about it because it hides behind an Interface. Maybe I can supply my own common_build() in make/xmingw/tasks.rb ? It turns out, the damn things behave just like I expected (I must have been reading the code, eh?) I added def common_build() to my tasks.rb and it did get called instead of the base class in make/make.rb. There is much suckiness but at least I can do that without trashing the other build types.

While I'm whining, how freaking hard would it to create some lists of at the main rakefile level that had the lists of gems and extensions to build? So you could drop bloopsaphone or add serialport. The make/make.rb hasn't been abstracted enough. NOTICE. There are two different compile paths for gems and extensions.

Have I mentioned my admiration for the Shoes rakefiles lately? This page is the only documentation for the rakefiles. How sad is that?

Jan 30, 2012

I continue to work on my make/xmingw/tasks.rb file. I copied the common_build() code and modified it to do the right thing for me. Almost. I had forgotten that extensions and some gems require that ruby is called (extconf.rb) to generate makefiles and of course any knowledge of the cross compiler to use is lost when the make files are created and they get built with my Linux compiler. There are two approaches to fixing this (google land doesn't have a lot of hints). I know which one I'll try first though.

I did however add some badly needed comments to my rake file mess to better describe what is happening or going to happen. I don't know why people don't comment their build files. At worst, the build takes 1 millisecond longer to process the comments. The pain of uncommented build files can be measured in hours.

Jan 31, 2012

Ah binject, my old friend. It's nice to see you again. Not really, it's just the first extension to compile and it's been a learning experience. Rather than damage normal builds I decided to make xextconf.rb files and call those from my rake files. Damage, you might ask? Oh yeah. extconf.rb and the makefiles it generates don't really understand cross compiling. It turns out the in extconf.rb file is a 'require mkmf' which effectively does a 'require rbconfig'. It needs to pick up the cross compiled Ruby's rbconfig.rb instead of the Ruby that is running the script. That's not TOO hard to fix, now that I know how. Passing in the includes directory so the mingw cross compiler can find the zlib.h? That took longer than I'd like but it looks simple now. At this point I can compile binject.c but it won't link because the the binject-i386-mingw32.def is defective. Who created that? The generated Makefile calls Ruby to build that two line file. Which Ruby is used? The cross compiled Ruby! Not the Ruby that is building things or works. Sigh.

mkmf is not well documented but I believe there is a way to tell it which Ruby to use in it's makefiles. In make file practice, once you have one from the bad Ruby, it's doesn't bother to create a new one. With a little bit of handholding (make clean) I was able to manually link a binject.dll. So, I just need to get my xextconf.rb to tell mkmf about the right Ruby.

It's actually decent progress. Slow progress but once I get the formula correct for climbing the first hill the other hills will be easier. Having looked at the code in mkmf.rb that I need to manipulate my way, I'm going to stop complaining about the Shoes rakefiles. Oh, there is much to whine about in the Shoes rakes files but once you stare them down, they aren't as scary as extconf.rb and mkmf.rb.

Feb 1, 2012

I managed to get binject built with my xextconf.rb and rakefiles. Time to use that knowledge on the remaining extensions ftsearchrt and chipmonk. Success! I can move on to the Gems. Similar issues with shoes/req/hpricot/ext/fast_xs, hpricot_scan, json/generator and json/parser. More Success. I'm a genius.

Then we have sqlite3. Oh my. Sigh. This validates my reason for making separate xextconf.rb files instead of hacking up the existing extconf.rb. On the bright side, I can learn something here about mkmf. Don't have a choice, either.

Shoes's hackers know I'm just a tiny bit away from the last of the C cross compiles and links and it's easy cruising after that. No. I'll get sqlite3 compiled, I'm not worried about that although the extconf.rb has lots of dealing with MSFT VC issues that should cause one to think hard. SQLite3 is weird. The Shoes gem may just be the ruby bindings to the sqlite3.dll/.so/dyld that is copied from the builders system and distributed when packaging shoes.

I suspect it's all about Windows. Rumour is you can can't mix dll and static libs or you go to hell. In a sane world, your Shoes.exe wouldn't require 20+ dlls in the same directory, most of them would be statically included into Shoes.dll. Of course that would be a different world than reality. SQLite3 and his VLC thingy would drive a free spirit like _why into a funk of pragmatism vs. responsibility of immense code bases and syncing shoes with them. Multiple versions of Ruby, Sqlite3, VLC, ever changing API's into them and a butt load of cheerleaders without the time or skill to help in the details. I'd go walkabout too.

Opps, back to reality instead of projections. I'm going to let the sqlite3 wait for a good think through. Some of the extconf.rb could be 'whatever works' hacks, some could be and some could be freaking brilliant.

Feb 2, 2012

I understand more about what Shoes is trying to do with Sqlite3. First, I had to download and cross compile Sqlite3 and install it in my Mingw deps library. I chose to modify a couple of xextconf.rb (binject, sqlite3) and the calling tasks.rb to be all little more mkmf friendly. I can actually compile the Shoes/sqlite3 .c code. Linking took a bit more effort but that too works. So, I've built all the gems and extensions and the Shoes code. I can now tackle the Builder task of copy_deps_to_dist. That should be fun.

But first I'll git add all my xextconf.rb files. So far, I haven't changed a bit of Shoes c or ruby code or the normal rakefiles. Everything has been in new directories and farking with new files.

Feb 3, 2012

Copy_deps_to_dist. Seems simple. Mostly. I had to replace some rbconfig derived values with my own. That make/xmingw/dlls file could be more useful in several ways. It should be built by a rake preflight task that verifies you have all the dlls and headers before you start compiling anything Shoes related.

I was caught with libungif. I don't have a dll. I'm not surprised. That was a strange autoconf build. I'll have to fix it. There's also some duplicate #defines complaints arising out of jpeg (I suspect I have confused headers (like more than I want in two many directories) and I worry about that winhttp.h that I borrowed from Wine. I expect another round of confusion when I get to the stub: task of building against winhttp.dll and nsis code but that's for another day.

Libungif is a mess. I can't get it to make a dll and its seriously unique in the world of OSS. For now I just copied the dll that came with the Shoes3 Windows. Now I'm missing portaudio. Or I don't realise what I'm doing. Roger that. I just dug a hole by copying the missing dlls. On the bright side Sqlite3 uses the same libtool that ungif does but now has a new name libsqlite3.dll so if that libtool worked, then ungif's should too. Right? Right? I'm also missing libssl32.dll in my deps. I don't expect cross compile love.

I'd better start a list of deps that I've ignored, just to get things to almost run.

1. Libungif -- PITA to cross compile. Not Done but their was a dll in the gtk bundle.
2. libportaudio --
3. libssl32 -- PITA to cross compile. Not Done. Watch out for version numbers.
4. libeay32 -- part of libssl32 compile. Interesting Patches here.
5. readline5 -- Is this really needed or is it a Ruby thing that probably isn't needed by Shoes? It also fails to link. Probably fixable but I suspect the presence of the dll is good enough for Shoes.
6. libiconv2 -- Builds with my crosscompile.sh script and why is the source code 1.14 and the dll 2? Opps. It didn't .

That's the missing dlls from the static make/xmingw/dll file. Which ones are really needed to be build or can they just be copied or they have different names (like sqlite3 did). I don't know.

Feb 4, 2012

I cleaned up or attempted to get some missing deps built from source. See the list just above. I'm just going to copy the libssl32,libeay32,readline5 dlls from a working Shoes. Time to remember how I got here. At the beginning I download a few deps like libungif and tried to build them. Then I downloaded gtk+-bundle_2.24.8-20111122_win32.zip which had a lot of shoes dependencies (.h. and .dll) and then I compiled some others (ruby 1.9.2 in particular) that were missing from the gtk2 bundle.

Ideally I would download the source code for all the deps and cross compile it. Maybe some day the world will be perfect and I'll have a perfect cross compiling environment. (I did run across a web site that has scripts for that - they start with building their own gcc to do the mingw compiling and a long list of deps and apps that the download and cross compile. That would be fun if you're fan of cross compiling). Me ? I want to get the next phase of building Shoes which creates the exe. The rake file :build finishes.

OK. I should have an exe somewhere right? Something I can feed to wine? Nope. There is a shell script /dist/shoes that might work for mingw/msys users on Windows to start up and test their new Windows shoes. Useless to me. FWIW, running that shell script in Linux is good way to test your Swap partition and patience. Type your ^C in fast or wait and wait. OK. Good to Know. Don't run that. Call that an epic fail. I think Wine should be able to test a Shoes.exe but without an exe that's going to be difficult.

That fail allows me to move on to the install task which amounts to getting NSIS to run in Linux and build the Shoes.exe that will be installed on users system (or downloaded for script packaging or later at net install -the later problem I hope to fix). Maybe I can get NSIS to build an .exe that Wine likes. But first the NSIS stuff. At first glance the language in their configure scripts resembles some kind of embedded FORTH.

Turns out nsis 2.46 (the latest) is available in Ubuntu synaptic/apt-get/apptitude. That makes things easier. Whether it includes or requires scons is not known. It also adds a few files to my mingw cross compilers directories (.h files, mostly).

A bit of fooling around in my tasks.rb and after a rake -f crossmingw package I have:

ccoupe@twb:~/Projects/shoes$ ls -l pkg/

-rw-r--r-- 1 ccoupe ccoupe 5497691 2012-02-04 23:12 shoes-0.r1527.exe

In theory, that will install Shoes when a Windows user downloads it. If it runs at all, it will not, not, repeat not package windows scripts for net install. The 5.4MB size is about right. It needs to be downloaded and installed by patient Windows users. FWIW (a lot?) the install runs in Wine and asks the right questions and the button clicks install it somewhere in Wine. That just means the NSIS code is Wine compatible but it's a good sign that NSIS works mostly OK. I've uploaded the file to my website as http://www.mvmanila.com/public/shoes-0.r1527.exe That file does install in Oraclebox VM XP2 but I can't tell if it's my shoes running or older shoes. I'm not that Windows savvy. Note to self, adding build and target info at the bottom of the splash screen would be a good idea.

Feb 5, 2012

OK, the shoes installer I distributed did not work. Oh, the installer part (NSIS) did fine but without shoes.exe in the payload there is nothing to run. You may remember my wondering about that. It's pretty obvious where shoes.exe is linked, at make/xmingw/tasks.rb:make_app(). I fail to see how a real Windows user with Mingw/MSYS would fix it up when making the installer but that's not my problem to solve.

So I fixed that code and now I have a dist/shoes.exe and Wine whines about missing dll's. Evidently that monster gtk bundle built some new dependencies in there . I was missing libfontconfig-1, freetype6, libpng14-14,libexpat-1 and intl so I added those to the make/xmingw/dll file, rebuilt shoes and I have a shoes.exe that wine doesn't complain about like it did before. Sadly it doesn't seem to do anything graphical. No error messages. That is unlike Wine's complaint about the normal shoes.

Sadly, it does nothing in Windows either. Installs fine but does nothing when asked to run so it's comparable to Wine ;^) Good to know that Wine can help me debug. Wine should at least put up the start screen like it did before. Something is not correct with my Shoes.exe.

I wonder if main.c and appwin32.c are confused about about things? There is a lot of #ifdefs that could go wrong if they think the target system is the build system. That looks clean at first glance. It could also be a ruby linking problem. In Linux, with a hand built Ruby you can get some odd startup problems. It might be time to learn about the Wine debugger (and not strip symbols)! Ick. Debuggers are better than hex coredumps though.

Feb 6,2012

Turns out that ashbb put together a little script to replace lib/shoes.rb for testing purposes and yes, it runs on Windows and even runs on Wine. Which means lib/shoes.rb has platform and/or ruby version issues. I'm guessing both. Since Ruby 1.9.3 doesn't work on my Linux box (rvm build), I'm not going to cross compile that. I tried going backwards to build 1.9.1p376 and it wouldn't cross compile win32/win32.c -- it complains about a (const *) *buf += len where buf is defined as (void *). There is a special place in hell for debugging casts and I don't want go back in there.

There is a 1.9.1p431 I could try (probably will). I did a comparison of win32.c between 1.9.1 and 1.9.2 and it's a massive amount of changes. Obviously 1.9.2 (or .3 if it worked) would be the better choice for debugging lib/shoes.rb. Presumably, 1.9.3 builds on 1.9.2.

Sigh. 1.9.1p431 fails with the same problem. Just for giggles, I might try to fix it in win32.c. No, I was right, it's the road to hell. Frankly if Red Shoes has a future. 1.9.2 and 1.9.3 need to be handled by Windows (cross compiled or native compile) and lib/shoes.rb and that means a lot of work. I can't step through the ruby with gdb or winegdb and wine tosses up a critical section error if I try a $stdout.puts from ruby.

Time for some rethinking.

(A few hours later) The normal Shoes lib/shoes.rb does all kinds of stuff - it parses the command line and dispatches to gem (hmm) and packager (hmm) and manual. Ashbb's code snippet doesn't include all that. That is a serious clue. For debugging a Shoes loading problem in Windows/Wine all that fun stuff has to go dark. If it still fails to get a splash screen (or the expected Wine whines) then I'll know the problem isn't gem/packager/manual mixins and mixups. In other words, make the smallest test case lib/shoes that works and then start addings things until the offender appears. Tonight, I'm guessing gem handling and load paths are misbehaving. But that's tonight. Might be something else tomorrow. And yes it would be nice to log progress or debugging messages to Wine from ruby.

So I started by adding the #requires at the top of the regular lib/shoes.rb to the dist/lib/shoes.rb that ashbb wrote that works. Fail. Sweet! Just 10 lines of Shoes.rb causes a fail. Comment out the requires one by one in a skillful manner and it

if Object.const_defined? :Shoes

require 'shoes/image'

end

Yes, it is shoes/image that fails. Comment out the require and the test script works. Leave in and Wine (presumably Windows) doesn't run. Why shoes needs to define constants in Object is a whine of my misunderstanding, but it does and shoes/image require fails in 1.9.2/Mingw/Wine. This finding does not mean those other commented out requires are good. It just means that require 'shoes/images' has a problem in 1.9.2. uncommenting require 'open-uri' causes a fail and that would certainly cascade down hill and it

Feb 7, 2012

Turns out it is open-uri and Ruby 1.9.2 and it might be fixed if I had used a more up to date Shoes code base since later lib/shoes.rb appear to have fixes for uninitialized constant Encoding::UTF_7. I'm going to commit my rakefiles to my git repo even though they are far from perfect. That way they can be shared.

Then I'll figure out how to get my repo synced with the current build. I'll also build a newer version of the exe.

I built a newer version of Shoes.exe, 0.r1741. It fails on XP and Wine. The good news? Both throw up a failure dialogue. In a sick way, it is progress since I've managed to duplicate ashbb's complaint that Shoes/1.9.2/Windows doesn't work. That means the cross compile is successful (in a sick way). My effort is no longer proof of concept. Cross compiling does work and Wine is a useful test bed because Wine reports an access to location 0x0 and lots of thread dump info just like Linux does when Shoes hits a machine level exception buy using a null pointer.

I'll have to recompile ruby and shoes and not strip the debugging info and hope winedbg reports more info about the offending caller. If it is a threading issue, then gdb or winegdb can effect the timing (as in it won't fail reliably). That too would be good to know, if it happens.

Feb 08, 2012

Looking at the stack backtrace reported by Wine indicates that the null pointer exception happens early in the Shoes startup which would explain why ashbb's trick of replace lib/shoes.rb doesn't work for my r1741.exe. It's not getting that far. Which is pretty odd since its the same ruby and dlls.

You can do winedbg shoes.exe or winedbg --gdb shoes.exe. The --gdb variant shows the the loading of the dll's and Wine believes something from libshoes.dll is incorrectly calling create_alpha_bitmap via an unknown path path to cursoricon.c (part of Wine or Windows). 693 cursoricon.c: No such file or directory

Feb 09, 2012

I'm certainly willing to believe that Shoes creates a cursor via Windows API's and there might be a missing file involved. But what happened in Ruby level Shoes code to cause this? Is this some sort of Ruby 1.9.1 vs Ruby 1.9.2 issue?. An interesting note. rake clean doesn't remove the compiled object files in shoes/native. Boo. They always get compiled, right? but...

Assuming the error message to mean that an icon file is missing or malformed lets look for what is in dist/static. Well, we have some new files in Shoes-r1741: app-icon.png and PKGBUILD. The first only appears to be Gtk/Linux thing and the later is some sort dependency tracker shell script or osx. Looking around, the Shoes.icns appear to be some more OSX magic. (note to OSX developers? where is this file created in the rake files?)

What would Shoes or Ruby change that fails to load an icon. 1) get the path wrong or 2) call a Ruby internal API that has changed and Shoes didn't. 3) Both.

Feb 10, 2012

Rat holes! You'd think I could detect them better after all these years. The cursor icon stuff appears to be a winedbg --gdb thing and less of a Shoe problem. Since it's the same ruby and dependencies dlls the only difference is the Shoes version that almost works and the Shoes version that completely doesn't work and there is likely to be very few C level code changes between those two Shoes since no one is maintaining the Windows C code (or the other platform's C code). That means the error is likely to be in the lib/shoes.rb code.

So I went back to debugging why the -r15xx almost works. Bingo! I may have copied the Ruby library into the wrong place. I may have copied the Ruby std lib from the wrong place in my rake file. I like this. It puts the problem back on what I did and I can fix that instead of mysterious rat holes.

Indeed. A make install in the xmingw/ruby-1.9.2p290/ does some additional contortions or the libraries. Remember, that when I do a make install the destination is /home/ccoupe/Projects/xmingw/[includes|lib|bin| I should not copy from the ruby directory, I should copy from where the (cross compiled Ruby was 'installed'). Damn you subtle differences!

I'm not saying that will fix everything, but it's a good place to start and its my Rake files that get fixed instead of the C or Ruby code. Even if it doesn't work, my rake files still need some clean up from the hacking. One step back, two steps forward?

Feb 11, 2012

I'm deep into where Shoes expects ruby standard libraries (say bigdecimal) to be and where the Rakefiles put them. I'm certain that if any of them need a binary so/dll and many will, it's not going work. My standard Ruby Standard .so live in i386-mingw32 and not in mingw. Did I mention I'm very confused?

Feb 12, 2012

I've got a Linux Shoes obviously and I assume that MingW Shoes will have a similar layout in dist/ruby/* and that turns out to be surprisingly tricky to do, thank you cp-r semantics. That can be sorted out.

This might be a good time to review what is in dist/ruby/. Only two directories gem/ and lib/. The gem/ contents include the Shoes gems: hpricot, json, sqlite3. Within those are the ruby and compiled C glue binary lib to connect with the .so (or .dll) somewhere else. That gems dir is created by the rakefiles and it appears they match between Linux and my cross compiled mingw Shoes. I'll move on.

The dist/ruby/lib directory is where all the Ruby Standard Library lives. Many are just Ruby (.rb) code but some need binary support libraries (so/dll/...) and those binaries live in a subdirectory. These binaries get copied from the Ruby location. Copied. For my 64 bit Ubuntu box it's x86_64-linux/. For 32 bit Ubuntu it would have a different name. For all the versions of OSX, there would be different names. For Windows there is ...?

mingw and i386-mingw32. The first is where the distributed Windows compiled MingW Shoes looks. The second is where my cross compiled Ruby is going to look. I think. But who knows? Shoes could tell Ruby where to look and I suspect it does. I suspect it tells Ruby to look based on RUBY_PLATFORM in which case mingw is the place to copy things to.

If you've assembled all that in your head, know thee that extension binary libraries like binject.so need to be copied in there by the rake files. Stop and ponder that. It's one reason you can't expect to distribute a universal Linux binary.

I should look at the Shoes start up code before choosing. Before then, I can get the rake files to build the dist/ruby/lib properly for either chose of arch names. I'm aware of the problem ahead and the knowledge I lack and I know where to look for the answers. I hack with knowledge. Getting shit to compile is not knowledge. Getting it to link is skilful. Getting it run? I'm going to fix getting my rakefiles to build and copy to dist/ruby/lib either way.

Silly me (or dumb shit me)? I have a copy of what is installed on a Windows system. In ruby/lib/ the subdirectory is i386-mingw32. Just like my build. Scurry to fix my rake files. I'm doing this with Shoes-r1527 (it's old, pre ruby 1.9.2), Pay attention, it going to get fun!

'ccoupe@twb:~/Projects/shoes/dist$ wine shoes.exe` gets me the expected error. as in nothing reported or working. Hey, failing gently is a sign of success in Windows. I put my windebug-lib-shoes.rb script in place of shoes.rb. I got the UTF_7 error, Expected. Good so far. I copied in a few lines of code from Shoe-1741 (?) that dealt with encoding Previously my Error trap went from UTF_7 whine to a 'file not found' on require 'open-uri'.

Now it reports no errors for the require or the encoding Oh holy shit! Dance for Joy! There is a a good chance my cross compile might work for Shoes-r1741. I'm not saying the job is done but I see the light at the end of the tunnel.

Feb 13, 2012

I added a few more requires from shoes.rb (r1741) and then I got an error thrown from require_relative 'shoes/cache', "code converter not found (UTF-16LE to ASCII-8BIT). Also of interest is an error message from Wine: Z:/home/ccoupe/Projects/shoes/dist/ruby/lib/i386-mingw32/enc/utf_16le.so: warning: already initialized constant UTF_16LE

Up to now I've been using the Shoes (r1527) code for testing. Time to switch to the r1741 and see if that error shoes up. So I switched. Compiles just fine of course and fails spectacularly with an impressive thread dump and backtrace. That old reference to 0x0000000. That means something in Shoes between r1527 and r1741 is wrong for mingw. Hmmm. I must ponder this.

On one hand, its winedbg without symbolic support. Ugg. Or use an old version of Shoes and reinvent all changes made to 1.9.2 or ?

Or do a diff on what changed from r1527 to r1741 in shoes.rb. Very little and most of it encoding related and conditional to linux or darwin which makes sense because I may be the first person to test this in mingw with a hand built Windows ruby 1.9.2 that didn't come from RubyInstaller. Note the caveats. Details matter. This not a good time to just hack till it works (unless I do). First I have to know what I'm hacking around in and I don't know this encoding thing.

I also dislike the negative logic of 'unless' applied to long strings of code class creating code. I know, all the kool kids do it and I'm a fuddy duddy that is not up to the demands of post modern Rubyism. Then again, I'm the one in here fixing yrz codes and adding comments in case some other non-wizard wants to help in the future. I think I'll get r1527 working in Wine and Windows and then the wizards can integrate those changes into the mainline when they get cross compile that and test it. I'd rather learn about Ruby 1.9.2 and encodings than deal with git.

Speaking of git. I committed and pushed my latest rake file changes that get the dist/libs correct enough.

Oh Noes! I peeked at shoes/cache.rb! That may not be sufficiently cross platform for Wine and cross compiling. I don't know why it's called cache either because that caching is not going on in their. There is something disturbing in there that I haven't accommodated. Who knew?

This is what happens, you get deep in the weeds, thinking about UTF encodings and ignored the obvious - look in the file that was pointed to as causing the problem. I'm not saying the Encoding errors I've mentioned don't play a part. A diff of what changed in in cache.rb from r1527 to r1741. Oh my. RUBY_PLATFORM is replaced with SHOES_RUBY_PLATFORM. If memory serves that was introduced for compiling multiple versions of OSX (10.5, 6, 7, 8). Whatever. Except I'm trying to limp r1527 into working and cache.db is throwing the error about encodings.

Feb 14, 2012

After inspection, there is no obvious reason why cache.rb should throw exceptions or even why it thinks something is UTF_16LE and needs converting and try to init a converter (.so) that's already initialised. I also have to consider that Wine could be confused.

FWIW, lib/shoes/shybuilder.rb will also trigger the same error. That file explicitly claims to need UTF-8. Oh my, there are other files in lib/shoes that set encodings at the top of the script. I really didn't want to know that.

This will be fun. I modified my debugging shoes.rb to report ENCODING along with the exception error message. US-ASCII appears to be the default from Wine/MingW/Shoes. On Linux (and I suspect OSX) UTF-8 is the default. That could explain the error, or be a red herring, but it's one hell of a RED FLAG. Now that I know some ruby files have the magic encoding to UTF-8 someone saw this problem before and hacked around it.

Feb 15, 2012

Sigh, the encoding holes gets deeper. Windows XP reports that ENCODING is US-ASCII so I can't blame Wine for that. Maybe Ruby could be cross compiled to do something else? It's all very confusing although attempts to educate me

As it turns outs there a lots of magic comments that set the encoding within the Ruby libraries. Some call for US-ASCII, some for ASCII-8BIT. What a freaking mess! Remember that the error message that it can't convert from UTF_16LE to ASCII-8BIT? I do. If I understand correctly all the bits of strings, script code or file contents get converted to [probably] UTF_16LE (or 16BE, 32LE, 32BE for some people). Each new string (code or file contents) is tagged with what it came from and they all get converted to whatever what everever is asked for. I don't have a UTF_16LE internal to ASCII-8BIT external converter or much more likely I haven't mapped US-ASCII to be the same as ASCII-8BIT (because they aren't same) but for display purposes they probably are. Confused? What about the UTF-7 error I hacked around.?

Feb 16, 2012

Grrr. Remember that UTF-7 problem and the Shoes fix in r1741 (create the class entry without any meat in the class). That is not a fix, that is a hack. My UTF_16LE error is a reminder that hacks don't fix the underlying problem when a newer Ruby arrives. There is no reason that UTF_16LE or UTF_7 should be needed to convert anything at this point in Shoes startup. Maybe if you require json (or net/imap for UTF_7) and who would require json at start up?

Oh! Might it be ruby/gems/1.9.1/gems/json-shoes-1.1.3/lib/json/add/core.rb? Why does Shoes need it's own version of json when json is part of Ruby 1.9.x ? Double double Grr. I know the answer. Some folks didn't want to live with the Gems or json versions included with 1.9.1 or 1.9.2 and they hacked it up in Shoes without testing on Windows (cross compiled or native).

Ironic aside: While researching the utf_16le problem with Google, I found a mailing list post by ashbb from Jan, 2011 reporting the same problem and that was about the time he stopped working on Red Shoes and put most of his efforts into Green Shoes. I think I know why.

I've replicated his error findings and I'm not happy about how far I'll have crawl back down the Shoes versions to find one that works in Windows. Probably back to Raisins and Ruby 1.8.7 and I just fix things to download Raisins from new websites.

Feb, 17, 2012

I whined to the Mailing List. I do that when I want to give up. One of these days, I really will. However Steve dropped a hint that caused me to step back a bit or pop the stack a few levels. That's what you need when you're in the maze and angry.

I think the problems is the '-' and '_' as later Shoes hacks almost accommodate. rgrep'ing UTF_16LE and UTF-16LE between MingW and Linux suggest that ruby 1.9.2 is not internally consistent with encoding names and the better hack would be to create an Encoding alias for whichever -/_ variant is missing and do it up top where UTF-7 gets faked out before any 'requires'

I'm not saying that will fix anything but it might and it's worth a try and it's something I can do.

Feb 18, 2012

I'm learning, slowly. Generally, Ruby Encoding names don't use underscores. UTF-8 not UTF_8. There is no Encoding object named UTF_16LE. For some reason there are Constants:

irb(main):010:0> Encoding::UTF_16LE => #Encoding:UTF-16LE irb(main):011:0> Encoding::UTF-16LE SyntaxError: (irb):11: syntax error, unexpected tCONSTANT, expecting $end from /home/ccoupe/.rvm/rubies/ruby-1.9.2-p290/bin/irb:17:in `

' irb(main):009:0> Encoding::UTF_7 => #<Encoding:UTF-7 (dummy)>

and a few ruby libs like uri and cgi use the Constants to get the encoding objects. I suspect they use the objects for comparison purposes. Perhaps a shit list of things to not use for transcoding uri and cgi things. Just my guess.

For Shoes purposes those Encoding::constants have to exist or some requires will fail. The first hack in Shoes.rb does that by creating an empty subclass with the _ name (say UTF_7) within Encoding if its not a defined constant. This assumes that Encoding.new("UTF_7") will create the Encoding::UTF_7) constant. Perhaps it does. Assume it does because it is good enough for loading 'open-uri'. What would the Encoding::UTF_7 constant point to? Nil or your empty class? UTF-7 is a dummy encoding. It's not supposed be used by regular people. I suspect there is some weird Ruby history with UTF-7 or UTF_7. But, the first hack is good enough for open-uri. Just don't write an IMAP client in Shoes or a Shoes http cgi server until you understand UTF-7.

But what about UTF_16LE and the others in the first hack? The correct class probably exists and the constants map properly.

Or, as my tests suggest, I still don't understand what's happening.

Feb 20, 2012

It took me a while to build a shoes.rb replacement that has decent enough support for multiple error messages. If your have a Ruby typo or syntax error, neither Wine or Windows will give you an error message. That makes debugging the Encoding issues more difficult than they already are since I typo. Also I was deep in the woods trying to understand the shoes.rb encoding 'hacks' My little shoes.rb debugging script just lists the Encodings at start up. I copied to my Linux version of Shoes. Big difference! In Mingw/Wine/Windows/Shoes, there are only 3 encodings, ASCII-8BIT, UTF-8 and US-ASCII loaded at start up time and US-ASCII is the default. In Linux/Shoes the default is also US-ASCII and the next 3 on the list match Windows. And then the Linux version lists all large number of available encodings including the offending UTF-7, the UTF-16LE and so many more.

My cross compiled ruby has all those .so files that Linux does so why are those encodings not available? Shoes or Ruby Problem? I'm guessing Ruby on Windows from Rubyintsaller or cross compiled doesn't include them it its list of font. I do remember building Ruby and it did compile the encodings and took some clock time to do it. Might there be a setting in the Ruby autoconf stuff that tells Windows Ruby not to use the full set of encodings, even if they compiled them. For a cross compiled embedded Ruby, that would not be unreasonable -- keeps the memory footprint low unless explicitly called for.

It could also be a rbconfig.rb setting (which is created when compiling ruby). Shoes.rb shouldn't have to hack around it with platform specific Monkey patches. It's possible that the Shoes C init code fiddles with or doesn't accommodate the Encodings directories in a Windows/1.9.2 friendly way. That would be the first place to look. On the bright side I now know of 3 possible places where Encodings go bad.

Feb 21, 2012

Here is how to confuse problem solving: Instead of seeing what went wrong with encodings in Ruby 1.9.2-p290 and the cross compiled Mingw of the same version. I installed 1.9.3-p125 on the Linux box using RVM. I'd read the CHANGELIST and there were some-things in there for Windows. p0 of 1.9.3 didn't install (yaml stuff) but p125 does and Shoe/Linux compiles and runs (I didn't do a lot of testing). I'm going to cross compile 1.9.3-p125 for MingW. Not because I believe it will solve my encoding problems or thread dumps. It might but I don't expect it too. It will allow me to see if I can find the reason to Windows/Ruby/Encoding problems.

In the cross compiled environment make ruby only builds the default encodings. That may also be true of the native compile. That makes sense. The make install task took a long time on native linux for 1.9.3-p125. It uses Ruby (fakeruby and/or miniruby) to do some heavy lifting for installing). Before I do that and potentially pollute the xmingw deps with crappola, I'm going to look at what the install task does for ruby 1.9.2 where it gets to the extra encodings. Compiling ruby by hand or by rvm leaves a lot of config files, makefiles and logs behind. I'm going to pop the stack on 1.9.3 until I understand what's going on and find the reason that MingW targets don't get the full set of Encodings in 1.9.2. Still, it's good to know the cross compile works. Would make install work better in 1.9.3 instead of 1.9.2?

It might be a good idea to move the Rubies out of ~/Projects/xmingw/[include|bin|dir] and fix my rakefiles to point to different rubies since the bulk of ~/Projects/xmingw isn't going to change. Decouple the deps and the ruby versions. Maybe ~/Projects/xruby[1.9.1|1.9.2|1.9.3] Those would be the prefix-dir for make install to install into or compile against so the would each have include/bin/lib/. Confused? Does it fix the Encoding problem? Not at all. But it would limit the damage I could do. Is that a cross compiler_ rvm like_ thing with additional perils? Yes.

But, If I'm going to have to deal with multiple Rubies and I don't see how I can't, I'll bite the bullet.

A few minutes later: I closed all the old terminal windows and started a new terminal. I mv'd the ~/Projects/xmingw/ruby-1.9.2/* to ~/Projects/xruby/1.9.2. cd there. I did a rake clean. Instead of make ruby I just typed make. It barfed way down the line because I had not set my rvm ruby. My default is 1.8.7 . Oops, my bad. Switch to 1.9.2-p290. Make clean and make (not make ruby). I watched it build all the encodings. Hmmm is make ruby; make install different from make; make install? I think it might be. I can't tell until I remove the ruby artefacts in ~/Projects/xminw and rebuild Shoes after fixing the Shoes cross compiling rake files. It could go all twisty passages again, count on it. But I feel I'm getting closer.

Feb 22, 2012

I've been reviewing all the 'might not work' comments in the above blather. I'm guessing I found one or two of them.

I'll note that there might be some issues with the libiconv which I appear to have compiled. So, I've got another possible problem. But first the rakefiles changes to separate ruby from the deps. Sigh. That broke my fragile xextconf.rb. Maybe I'll love extconf.rb and mkmf someday. Just not today.

Feb 23. 2012

OK. I got binject to compile and link with a tweak to my xextconf.rb. Json and Hpricot didn't need tweaks and then I have sqlite3. That's ones a bit tricky since the freaking msvcrt-ruby191.dll lives in {ruby}/bin instead of {ruby}/lib and mkmf isn't that smart about cross compiling. God bless Windows. Fixable, I'm sure.

Feb 26, 2012

Or not fixable. mkmf.rb has many limitations for cross compiling, lack of documentation being the biggest limitation. I don't waste any more time. I'm going to proceed by documenting the requirement that no matter where your version of cross compiled Ruby lives, the msvcrt-ruby191.dll needs to be copied to your cross compiled /bin That's just something you have to do if you go from 1.9.2 to 1.9.3. (or the other way).

And still the Encodings issues exist. Time to look into how iconv was compiled (the README is a bit opaque). I'll copy the dll from the Windows Policeman 3.0, it is different from the one I compiled. How different. Lets start with the name libiconv2.dll (old version) and libiconv-2.dll (my version). Mine has another 300KB of stuff. The older version didn't fix anything. Even more odd, removing all the libiconv*.dll in dist/ doesn't throw an error in Wine. Neither one was invoked from Ruby or Shoes

Although my compile of Ruby built all kinds of Encodings, it might be possible that it never bothered to you know, like maybe link against. Perhaps it didn't find an libiconv* it liked (mine) so it ignore that and just included the basic UTF-8, US-ASCII and ASCII-8BIT (they don't seem to need encodings). Time to build ruby 1.9.2-p290 again and pay attention. But first, copy that older libiconv2.dll to ~/Projects/xmingw/bin and rebuild Ruby and Shoes.

No Change. It's looking more and more like a Ruby compile problem.

Feb 27, 2012

Aha! I found that require 'tmpdir' will throw the UTF_16LE error on Windows ruby and its probably a known bug in core Ruby. As well as shoes/cache.rb, it's needed by shoes/shy.rb and some of the gem handling code. There are several possible workarounds:

1. Get all the encodings loaded in Windows ruby. This too may be an error in Ruby (MingW files).
2. Monkey patch the Dir.tmpdir with my code and comment out all requires of tmpdir. Unpleasant.
3. Provide an alternative tmpdir.rb and copy it to replace the normal Ruby file or load it before the normal ruby file. Maybe do a UTF-8 secret comment at the top or ASCII-8BIT comment. Still a hack.

Either way, I need to look at Ruby's Dir and tmpdir code to find the source of the conflict before I hack or even fix it. So I looked at tmpdir.rb (line 15 is suggested to be failing in some forums). I can see how there might be some issues. It explicitly requires 'etc.so' and explicitly does nothing on a LoadError rescue. Curious. I have an etc.so. (both Linux and cross compiled) and loading it with Shoes/Wine appears to not trigger an exception. I'll de-construct line 15 to find where that UTF_16LE arrives.

Bingo! When I use Etc.systmpdir as a var I get the UTF-16LE encoding error. Shoe's default encoding is US-ASCII (fallback ASCII-8BIT). I haven't looked at the C code of etc.so but I'm guessing it picks file encoding based on LE and BE and 16/32 tests and assumes there is a translator 'up top'. In the case of MinGW compiled Rubies, there aren't those Encodings. Not so much a Ruby core problem as a MingW Ruby build problem. It compiles the Encodings but it blind to them when running. There is no amount of Shoes.rb hacking that will fix that.

Feb 28, 2012

As I expected, ashbb created a monkey patch that at get Shoes running on Windows with modern Rubies. That's fine for testing what else works in Shoes/Windows but it's not good enough to release a new shoes. Better to create another tmpdir.rb called 'wintmpdir.rb' for example, that has the 'hack'. Then modify the Shoes rake files to switch wintmpdir.rb for the normal tmpdir.rb after it's copied the Ruby libs to dist/ruby. Require tmpdir happens in a lot of places (notably in gem handling). This scheme would also document in the rakefile that tmpdir has been given special handling and it doesn't break Normal ruby on Windows, only Shoes's version of Ruby and when the real problem is fixed it can be undone with a #

I'll do that to my version of Shoes. I created a patch dir in shoes, copied in the tmpdir.rb and I'll fix the rake files to overwrite the one from Normal Ruby. I need to poke around in the rake files for another reason. Oddly enough, my wintmpdir.rb doesn't work like I thought. It works well enough for shoes/cache.rb the first require and something dies on the second require. I looked at ashbb's hacks in shoes.rb dealing with encodings. NOES !!!

The real problem is how Shoes and Ruby and the Platforms deal with encodings and all the hacks in various shoes.rb variations. You can't create encoding constants that point to nothing (UTF_7, or UTF_16LE e.g). If your ruby is correct, you don't have to and if your Ruby is incorrect it's always going to be incorrect and might even become more incorrect in the future. Fix it! Patches to hacks on patches of hacks.

I'll grant you my hack to wintmpdir aka tmpdir.rb screws up on the second require. But it too was only a hack and the real problem needs to be solved. **Stop hacking !!! Fix the real problem before it get worse. ** No more monkey patches into Encoding to fake it so they load. Fix it so the god damn thing work!

Feb 29, 2012

The best way to undo hacks is to understand what was hacked. Wine complains that a Ruby Constant is being Z:/home/ccoupe/Projects/shoes/dist/ruby/lib/i386-mingw32/enc/utf_16le.so: warning: already initialized constant UTF_16LE. The first init was the hack in shoes.rb. But wait. The Wine error message is from the utf_16le.so. That means MinGW Ruby can find and call/init on the encodings. Ruby knows where they are, it was called and it complained about the hack class/constant. Why MinGW Ruby Shoes.rb needs the hack is called into question. Also why it can't enumerate them in a list.

I went searching in the ruby source code for where the error message string is built. That's transcode.c:rb_econv_open_exc(...) Who calls that? transcode_loop(),econv_s_search_convpath(),econv_init(). The latter seems to also emit that Wine already initialized message so that is a good place to start my quest for understanding. The first executable block of code is:

if (rb_check_typeddata(self, &econv_data_type)) {

rb_raise(rb_eTypeError, "already initialized");

}

Right away we can see some huh? self is the ruby object. &econv_data_type ? My C skillz think that &econv_data_type is the address of a static variable (struct probably or more likely an array or list of structs or VALUES )

Two things to look into. What is econv_data_type and who calls econv_init? I suspect the init will be more informative.

Mar 1, 2012

Unlike Shoes, the Ruby code has comments and the comments ahead of econv_init provide lots of information. One of which is that it's the init method of Encoding::Converter.new. Might be time to grok the essence of the entire file of transcode.c. I'm not certain the problem is in here but a good clue might lurk in there. There are several layers. ec_xxxx is the Encoding Class implementation. rb_xxx is the lower level stuff called by ex_ functions. Anything defined in various .h file and implemented here is callable from any ruby code. It's confusing place. Don't try to understand everything.

I think my bug is really in the Ruby Makefiles for MinGW but it might be in Shoes init or Ruby init code that is conditioned on MingW. My approach is bottom up. This is not the fast way to fix a problem. It is a good way to understand a problem without a debugger.

A curious mind might consider setting up new Encoders in shoes.rb (from say UTF-7 or UTF-16LE to ASCII-8BIT, or UTF-8) and see what the error messages are. That appears to be an annoying path in Wine with the limited debugging ability of my hacked shoes.rb. Did you know rvm compiled Rubies don't have readline support for irb? . That's enough fail for one night.

Mar 2, 2012

Transcode.c is not where my problem to be solved exists. Time to climb up the stack of abstraction. Somebody has to trawl the encoding .so. It was a lovely visit inside. Like I said before, bottom up exploration is slow but sometimes you can't see the obvious until you know enough.

Mar 3, 2012.

I don't have a lot heart for the problem tonight. I'll note that in C definition of the Encoding.Converters is transcoder_table = st_init_strcasetable(); . In transcode.c it's only referenced, not defined. That's probably worth looking for. Looks like it is defined in st.c, kind of not really. st.c implements hash functions and defines several internal ruby hashes and it just returns the address of that table. It's all very Ruby dynamic so it's callers that add to that hash is where I want to find the scan of available Encodings. This is the point where static bottom up analysis requires some very careful thinking and backing up. For instance encoding.c calls it as does marshal.c. Perhaps one of them looks in the enc/ so the .so

marshal.c: Interesting clues. It appears Marshal is a Ruby module/extension more or less calls yaml. Not the place to look for inits of the system encoding stuff.

As I expected, encoding.c is the C code for implementing the Ruby class Encoding. No doubt it calls the other stuff in transcode and st.c. On a whim, I did a search for WIN32 in that file. There are two places where the _WIN32 C constant is used. Similar but different. Pay attention. I suspect I have much to learn. One use of _WIN32 is in enc_set_filesystem_encoding(void) where it sets up the filesystem encoding default. That's what systmpdir() returns in etc.c which is called from tmpdir.rb which triggers utf_16le error later. Assume _WIN32 was defined when compiling Ruby. I don't remember it. I thought it was WIN32 and yes I'll have to look at the log files. If it is _WIN32, then some very interesting things are done to try to choose CPxyza encodings and then if that fails, ASCII-8BIT. Hello. If that code isn't compiled then there is another mess of ways things could fail on Windows. I'm not saying that is the problem but it surely is a high value clue. The CPxxxx look up would fail if the encoders are not loaded and it

rb_locale_charmap(VALUE klass) also has some possibility of getting it wrong for Shoes purposes but I'm guessing the money and the bug is getting the encoders loaded so there are CP1251/CP1252 encodings loaded in MinGW Ruby. I'm calling it forward progress. I just need to find where that fails Yesm. I smell a Wumpus approaching. Yes, I did have Kim-1 mother board with a home built power supply. I know Wumpus.

Mar 4, 2012

What a brand new day! Ashbb figured out that for Windows you just have require the damn encodings and he provided a simple, elegant method to do it. It causes some new problems for my cross compiled Shoes and as I've discovered, there is work to be done getting Linux to work with that. The handling of encodings is passed off to lib/shoes/encoding.rb to deal with (a wonderful idea, IMO). I've modified encodings.rb to be callable by linux and darwin (it does nothing extra for them, and that works for Linux).

Sadly, Wine pukes with a thread lock, even with the debugging shoes.rb when it uses the the encoding.rb. But it is progress. Kind of.

Mar 5, 2012

Maybe I'll move to Ruby 1.9.3-p125. Not because I believe it will fix my cross compile issues. It might, but I don't expect it will. I need to get near the same page with the cool kids. Get my version of Shoes at github near theirs. Or maybe I'll do nothing. I need a typing break. I've proven cross compiling is viable although difficult to debug and maintain. I'll update my shoes at github and go on a walkabout.

Mar 18, 2012

Time to mount up and fix something! I need to get the cross compile scripts working with Ruby 1.9.3-p125. I really need to go back and cross compile the dependencies that I cheated by copying from ashbb's old Policeman distribution. I need to understand gettext and libiconv configure/make to do that. Uggh! Ick! But, it needs to be done. I also have to do my tax forms and that task has a hard deadline approaching.

I don't understand git enough, I'll have to fix that. But, first I'm going to get my Linux rvm up to date and working the way I think it should. Get the ruby's to use readline[5|6] in irb. My old version of rvm was 1.0.5 and it would build a ruby without readline support if the dumb user (Me) didn't have the readline5-dev and readline6-dev packages installed. It took a few head scratches to get rvm up to 1.10.3. Hint: $ rvm --head update Then you have to do $ rvm requirements and read what it says to install. I didn't have readline6-dev. That explains that.

I'll rebuild ruby 1.9.3 with $ rvm reinstall 1.9.3. Oddly enough, it downloaded and installed yaml-0.1.4 in my rvm tree. That explains another mystery and might be an issue that was discussed in Shoes. FWIW My 1.9.3 reports 'Pysch' when I require 'yaml' and print the constant YAML.

Mar 19, 2012

I've got a Linux rvm installed Ruby 1.9.3-p125 that works fine for my version of Shoes likes (well enough for my purposes). RVM complains when I sudo but I don't care about that as long as the right version of Ruby is used in sudo (ubuntu's 1.8.7 for su)

There is a syntax error in pack.rb to track down if you try $ dist/shoes -p That's not right.

I still need to cross compile 1.9.3-p125 and build the MinGW dependencies more properly.

Apr 11, 2012 Long Time, no update

No updates because 1) Shoes lacks C level developers so it's discouraging. 2) ashbb showed up and mostly updated the windows code. 3) Shoes developers wander around without focus - herding cats in a field of shiny toys to play with. Me too. 4) I need to get synced up with github and I'm avoiding that. 5) I upgraded the linux box to Ubuntu 11.10 in anticipation of a 12.04 release soon. Might as deal the nasty UI issues now. One could spend many more person hours getting that correct.

I also discovered a few things that might help the Shoes/Linux rakefiles or startup files and possibly improve the cross compiling experience. It's called pkg-config. I must have been busy when that rolled into Linux. That and some uname parsing could fix my hack in the rakefile find_and_copy() function.

It also has implications for cross compiling (the autoconf/configure stuff might need/create pkg-config .pc files)

But first, I gotta get the git repositories synced.