-
Notifications
You must be signed in to change notification settings - Fork 19
A Developer's Tour Through The Magic Packager
This page is under construction. Expect loose metaphors and incomplete paragraphs.
Remember when you discovered that "Twisty passages all alike" and "Twisty passages all different" were two different mazes in Adventure? Well, I remember feeling pretty stupid. Packaging and Installing Shoes are like the "Twisting passages" because they are related, in too many ways but they are different. Installing is what happens when you download Shoes for your system (or the user does it for their system).
First I want to establish some terminology. Packaging and Installing for some people might seem like the same thing. They are not. Not from a users point of view or a developers point of view.
Some things are done in a rake :task and some are done inside a running Shoes program. As a developer with all the source code and rake goodness, you can create an installation file. If you are running on Linux (Ubuntu 11.04 32 bit) you can create a Linux binary and it's install package that mirrors what your Linux looks like (more or less). It might run on Fedora X64 or Arch32 or Ubuntu Wascly Wabbit. It probably won't but you can try and it might work. If you are on OSX 10.6 then you might be able to create an shoes.dmg installer that works on 10.5 or 10.7. Maybe. Of course if you want to make a Windows install, you have to be running Windows.
When the brain trust at Shoes HQ decides to release a new version of Shoes this is what happens: One person makes the Windows install on Windows, one person does the osx magic on OSX and one person does the Linux. Yes one person could be a masocist. Lets call that developer person Judy Developer. Then all Shoes.exe, Shoes.dmg and Shoe.run are uploaded and the Shoes website is updated "Get Your Newest Shoes Here!" That process I call creating installers because that what it is. In the Shoes Rakefile, to do that you would issue a 'rake package' command. Confusion alert! That is not packaging a script (see below). There is a variation of creating installers that hacketyhack uses so it creates HacketyHack.exe, .dmg, .run which includes all of hacketyhack and Shoes and has to be done on each platform and the resulting installer uploaded to the website where "New Hackety downloads are here". That installer uses app.yaml which just substitutes Hacketyhack for the name Shoes and builds an installer and run on all three platforms and uploaded. OK, it's a tiny bit different than creating a Shoes Installer but not much different. Its fricking yaml. It overrides things in the rakefile (and not that much).
Packaging a script is different than creating an installer. First off, you don't do it from the rake file, it has to be done from a running shoes program. Shoes developers don't do it. Any user of Shoes can package a script, say its chickendance.rb and when God smiles upon them, they can create chickendance.shy, chickendance.exe, chickendance.dmg, and chickendance.run. Joe RandomUser can then upload those to his website and send an email "Mom, download ChickenDance from my website. Just pick from Windows, OSX or Linux and it downloads and runs. I told you I'd amount to something, Mom." That is script packaging and it is not install packaging. Except when it almost mostly similar, but different.
Now that I've drawn the line, they all share some common developer code in Shoes and that will trip up new developers if they don't realise the packaging has too too many meanings. Get ready for the next "how does that work?" deep dive. A packaged script (chickendance.rb in the chickendance.exe or ,dmg or .run) might include all of the Shoes binary files to run the script or it might include a 'stub' that downloads the 'Shoes binary parts' if you don't have them. It's Joe Random's choice how he wants to package his script - include Shoes or download Shoes if needed. If Joe chose net_install then when chickendance.exe (or .dmg) was double clicked at mom's computer, her machine would download and install Shoes for her (after doing the Windows un-trusted authorisation dance of clicks) and then Joes chickendance.rb would start up in Shoes and Mom thinks you're not the fool you used to be because you got some mojo working.
What is in that "download if needed" file? Hold your head steady so it doesn't spin around. It's the same Shoes Installer exe./.dmg/.run that Judy Developer (remember her?) worked so hard to build (rake package and upload, for each platform). Yes Joe's mother could have downloaded the Shoes install herself (clicking past the Windows warnings of course) and then downloaded Joe's script and then run Shoes and then navigated to wherever she saved Joes chickendance.rb script and run it. Odds are Mom wouldn't do that and if your mom finds this web page, you're going to have to explain to her about the Judy person and why you are going behind her back to install things.
There is another feature of Shoes install or packaging/install that really makes a difference to developers because it permeates the soul, spirit and code of Shoes. A downloaded (or net installed) Shoes is as completely self contained as possible, and it runs in a sandbox - its not installed in System directories. No admin password is needed to install Shoes or to Package a script for other users. That's the goal or spirit of the goal or the best intention.
So packaging and installing are capabilities that can not be separated, they depend on each other and they depend on the website delivering the Shoes install at the time the script was packaged (include Shoes) or the Shoes at the website when the script is double clicked (net install)
What's in a Shoes install that is downloaded (one way or another)? Its binary C Shoes libraries and all the Ruby libraries (so/dll/dyld) and binary gem like extensions like sqlite3 and some Ruby scripts. Every thing in the developers dist/ directory. The Shoes start up code, both C and Ruby expect and keep everything in that sandbox that is a Shoes install. You can't load dlls or .so or gems that aren't in Shoes' directories.
All that stuff gets packed up into a platform specific container format. makeself for Linux, HFS/DMG, and something for Windows (zip?). See the Rakefile task 'package'. It includes an exe/app Windows/OSX binary into the container file to be downloaded. That exe/app is what is run when the user doubles clicks to install Shoes.
That bit of native code has to create the directories in the users directory, unpack the files in the download and put them in the proper place in the user's (not system) directory , install an icon on the desktop and whatever OS specific magic is needed to make the install double clickable. Then you the Shoes developer, put that up on the website for download. Exe, dmg will need to be done separately on Windows and OSX systems.
It's almost like what the rakefile does for creating a Shoes install "rake package" and script package are different - do not be confused by similar names). Instead of a rakefile to compile and massage binaries, the Shoes packager code takes the specified script/directory and bundles it up in a formatted file (tar, dmg, zip?) along with a exe/dmg/sh to be run when double clicked. Depending on options chosen by the user (A, B, C), they might include the shoes install with their script.
That exe/app in a packaged script is not the same as the one that used in creating the installer. That one has one task. Install Shoes. This one has to figure out if the downloaded file container it was in has Shoes. If so, install Shoes like the installer program would. If not, download the installer from the website and install that. Read that again until you understand the differences between these two exe/app's. One installs. The second one installs if needed (there might have been an install by previous downloads - yes there are dragons with that) and downloads and installs and then it use the install it finds/creates to run the users script. Two different exe or dmgs or .sh. Similar but quite different goals.
Oddly enough, in the Shoes source code they live in separate directories because they serve different goals. Once the packager used code is compiled to exe/app and is working and it can download the installer from the website if needed, it doesn't need to be compiled again no matter how Shoes changes. Those bits of compiled code live in static/stubs and they'll be part of any Shoes on any platform.
In platform/{msw|mac|nix} is the source code to create the exe/app/dmg in static/stubs but it also contains native code needed to create the installer bit. Confusing? Yes, but I'm not sure I'd do it better if starting from a clean slate.
You keep talking about Containers. What are they? It depends on your Operating System. It's file with a bunch of other files and directories. It might include application code. It's got an Shoes icon and its double clickable so its an application or will start one to process the container. Zip or dmg or tar ball.
This bit of code lives in the lib/shoes directory. Looks like a gem. IT USED to be a gem at RubyForge. Why is it not a gem now, you might ask? Why would Shoes have a private copy? For one, it's an ancient unmaintained gem. It's a few minor keystrokes different that the latest (but old public gem). You could say it's an unmaintained gem. I would agree. That's why its in the Shoes code. Someone has to maintain it.
It also has some coded added to deal with Windows and Shoes. It's not the ancient, unmaintained Gem.
Minitar is a Ruby implementation of tar. It creates tars or extracts tars. You can't depend on windows having a tar to call, so it's built into Shoes. As tar implementations go, it is minimal. It can handle directories and files with the posix info -name, dates, some permissions. Minimal.
A .shy file is a directory or just a single file packaged up with minitar. Since the person using the .shy has Shoes and Shoes has minitar the directory and files can be extracted. There appears to be some Hacketyhack meta data added to the .shy.
Minitar is also used to create packages for Linux. At this time, it fails because Minitar can't copy or create hard or symbolic links and the Rakefiles that create a Linux Shoes installation, create two symbolic link to shared libraries. Might be easy to fix in the Rakefiles, but Linux really is a build from source proposition so who cares?
The Ruby code in minitar.rb is a bit not completely fricking obvious and somewhere in there it treats the file sh_install differently if its Windows. I think it sets execute permissions in a Windows unique way and started to gag at the implications and stopped reading.
From the outside, the API that is called from pack.rb to create a .shy makes perfect sense and that's where you should start in understanding minitar - look at the calls into it, for what purpose.
Caution! I haven't figured this out. There are two paths into this code. One for creating Windows things of which I know little and another path for creating OSX dmg's which I kind of used to know. Start with the pack.rb code and it's pretty obvious what the binject code is supposed to do. Create in memory (or disk) images for the Windows or OSX containers (see I said it again). Disk images means it has to mirror FAT/NTFS/HFS/whatever pieces parts which may need the help of zlib compression library calls. There are C stdio disk ops in there which might be a thread problem for Ruby. There maybe call backs from C into Ruby to report progress to a Shoes progress bar. Those can be death to threading in Ruby 1.9.x. They might be commented out or they might still lurk.
Binject has two faces. One to create a Windows exe (Binject::EXE) and Binject::DMG to create a OSX .dmg. There are three methods to note, new, inject, save . New creates the file and in memory structures. inject inserts things (a script, a shoes exe, some meta data like windows resources or plists. Binject is not OO. Do not expect EXE::inject to do the same thing as DMG::inject. Two different critters called from two different parts of lib/shoes/pack.rb
Please, please remember, we are packaging a Shoes script (a ruby text file) plus any static files, images, directories) into something that can be downloaded by someone else. We can do that from Windows, OSX or Linux for Windows, OSX and Linux. We need to include enough native executable into the users system to install Shoes from a download or get and install Shoes from the downloaded file. The two choices are 'Install Shoes if needed' aka net-install and 'Include Shoes'. Pay no attention to "Include Shoes with Video" because that's not going to work for everyone.
There are some Windows resources to set up and include in the container. Binject::EXE.new() copies in 'blank.exe' from static/stubs to start the container.Then pack.exe injects a resource, SHOES_FILENAME which is the script name string without an extension. It's how Windows rolls. Then it injects a resource SHOES_PAYLOAD which happens to be the users Shoes ruby script. Next up, pack.exe calls pack.pkg("win32") . That bit of code downloads the Shoes Window install and returns the file handle or if the user packaged a net-install, the file handle is nil.
In Raisins (Shoes2, which worked just in pack.exe() ) if the file handle is not nil (i.e. Shoes is to be included with the script) then another resource is injected into the container - SHOES_SETUP, which contains the downloaded Shoes from the website - yes that makes for a large container). Either way, net-install or Include Shoes in the container', then exe.save() is called. It worked whether you call it from Windows, OSX or Linux.
In Policeman, we went backwards (aka broke things). If the user chooses to package "with shoes" it works just like above. and the exe works just like Raisins did. If the user chooses "net install" for windows all of the above exe.new() and injects() are thrown away - without closing the file handles (boo) and it shells to run a Windows shoes-stub-inject. exe. That is a fail for Linux and OSX and possibly even on some Windows and can only be done on a Windows machine. As best as I can tell shoes-stub-inject.exe does the same thing as the Raisins code did.
Eric Watson [email protected] found a _why email that explains things in less words than I use. I thought I knew what was going on and that email says I'm not completely mistaken. Eric is also planning to fix the OSX bugs so maybe I should try my hand fixing the Windows regressions on the packager functionality. Except I really hate typing things into the Windows terminal and all the C: and \syntax. If I could build on Linux that would be could. Turns out
- You can cross compile to mingw on linux.
- You can get the Nullsoft nsis installer to package up a windows .exe from linux.
- You might be able to test with Wine.
Even more interesting links . If wxWidgets and C++ and do it, Shoes should to. Brave people with OSX/intel could probably do the same.
To be discovered.