Memory errors on Showing, Disk Space runs out on Create

[StephenBrown2]

This may need to be two separate issues.

1. As a normal user, creation of a blueprint shows no errors, but showing it causes a Memory Error

As normal user:

blueprint create production-test
# [blueprint] using cached blueprintignore(5) rules
# [blueprint] searching for configuration files
# [blueprint] searching for APT packages
# [blueprint] searching for software built from source
# [blueprint] /usr/local caused EACCES - try running as root
# [blueprint] searching for Python packages
# [blueprint] searching for npm packages
# [blueprint] searching for PEAR/PECL packages
# [blueprint] searching for Yum packages
# [blueprint] searching for Ruby gems
# [blueprint] searching for service dependencies

As normal user:

blueprint show -S production-current
Traceback (most recent call last):
    File "/usr/local/bin/blueprint-show", line 56, in 
        b = blueprint.cli.read(options, args)
    File "/usr/lib/python2.6/site-packages/blueprint/cli.py", line 70, in read
        return blueprint.Blueprint.checkout(name)
    File "/usr/lib/python2.6/site-packages/blueprint/__init__.py", line 61, in checkout
        return cls(name, commit, **json.loads(content))
    File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
        return _default_decoder.decode(s)
    File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/usr/lib/python2.6/json/decoder.py", line 336, in raw_decode
        obj, end = self._scanner.iterscan(s, **kw).next()
    File "/usr/lib/python2.6/json/scanner.py", line 55, in iterscan
        rval, next_pos = action(m, context)
    File "/usr/lib/python2.6/json/decoder.py", line 183, in JSONObject
        value, end = iterscan(s, idx=end, context=context).next()
    File "/usr/lib/python2.6/json/scanner.py", line 55, in iterscan
        rval, next_pos = action(m, context)
    File "/usr/lib/python2.6/json/decoder.py", line 183, in JSONObject
        value, end = iterscan(s, idx=end, context=context).next()
    File "/usr/lib/python2.6/json/scanner.py", line 55, in iterscan
        rval, next_pos = action(m, context)
    File "/usr/lib/python2.6/json/decoder.py", line 183, in JSONObject
        value, end = iterscan(s, idx=end, context=context).next()
    File "/usr/lib/python2.6/json/scanner.py", line 55, in iterscan
        rval, next_pos = action(m, context)
    File "/usr/lib/python2.6/json/decoder.py", line 155, in JSONString
        return scanstring(match.string, match.end(), encoding, strict)
MemoryError

1. As root, however (going by the recommendation of the earlier warning), creating eats up memory still and hogs a bunch of CPU still, and eventually runs out of disk space.

As root user:

blueprint create production-test-as-root
# [blueprint] using cached blueprintignore(5) rules
# [blueprint] searching for configuration files
# [blueprint] searching for APT packages
# [blueprint] searching for software built from source
Traceback (most recent call last):
    File "/usr/local/bin/blueprint-create", line 74, in 
        b = blueprint.cli.create(options, args)
    File "/usr/lib/python2.6/site-packages/blueprint/cli.py", line 32, in create
        b = blueprint.Blueprint.create(args[0])
    File "/usr/lib/python2.6/site-packages/blueprint/__init__.py", line 69, in create
        getattr(backend, funcname)(b, r)
    File "/usr/lib/python2.6/site-packages/blueprint/backend/sources.py", line 191, in sources
        _source(b, r, pathname, c.cwd)
    File "/usr/lib/python2.6/site-packages/blueprint/backend/sources.py", line 165, in _source
        shutil.move('tmp.tar', os.path.join(old_cwd, tarname))
    File "/usr/lib/python2.6/shutil.py", line 264, in move
        copy2(src, real_dst)
    File "/usr/lib/python2.6/shutil.py", line 99, in copy2
        copyfile(src, dst)
    File "/usr/lib/python2.6/shutil.py", line 54, in copyfile
        copyfileobj(fsrc, fdst)
    File "/usr/lib/python2.6/shutil.py", line 30, in copyfileobj
        fdst.write(buf)
IOError: [Errno 28] No space left on device

But I can see no difference in file system usage from before and after. I didn't check it while it was working, I was focused on memory usage. Where does the temporary file get created? There is 4.5G free in my /tmp partition, should it be more? What is it doing that requires so much space?

[StephenBrown2]

I'm on CentOS 5.7, running Python 2.6 and blueprint grabbed from GitHub on March 24th using:
git clone git://github.com/devstructure/blueprint.git blueprint-src
cd blueprint-src/
git submodule update --init
make && sudo make install PYTHON=/usr/bin/python26

The VPS I'm on has 8 GB of ram, the aforementioned 4.5GB of /tmp space, and plenty of room on the rest of the partitions.

I'd assume that tarballing up the source files might be taking up that space, but I can't imagine it's 4.5 GB of compiled programs, unless it's trying to package up all of the files uploaded to the websites hosted on this box.

[StephenBrown2]

Update:
Reverting line 162 in 5990f3d to os.rename('tmp.tar', tarname) fixes the space issue ( I have at least 10GB free on /, where root's home is), but then a new error is shown:

blueprint create production-test-as-root
# [blueprint] using cached blueprintignore(5) rules
# [blueprint] searching for configuration files
# [blueprint] searching for APT packages
# [blueprint] searching for software built from source
# [blueprint] searching for Python packages
# [blueprint] searching for npm packages
# [blueprint] searching for PEAR/PECL packages
# [blueprint] searching for Yum packages
# [blueprint] searching for Ruby gems
# [blueprint] searching for service dependencies
error: 45109ada74b47cb49801ddb24df33b8b58239787.tar: does not exist and --remove not passed
fatal: Unable to process path 45109ada74b47cb49801ddb24df33b8b58239787.tar
Traceback (most recent call last):
  File "/usr/local/bin/blueprint-create", line 74, in 
    b = blueprint.cli.create(options, args)
  File "/usr/lib/python2.6/site-packages/blueprint/cli.py", line 39, in create
    b.commit(options.message or '')
  File "/usr/lib/python2.6/site-packages/blueprint/__init__.py", line 411, in commit
    git.git('update-index', '--add', os.path.abspath(filename))
  File "/usr/lib/python2.6/site-packages/blueprint/git.py", line 83, in git
    raise GitError(p.returncode)
blueprint.git.GitError: 128

Running blueprint list returns nothing (i.e. no branches have been created)

Running the git testing script from #54 returns these errors:

fatal: Not a valid object name production-test-as-root
git ls-tree: 128
BLOB:
fatal: ambiguous argument '': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions
fatal: ambiguous argument '': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions
0
git show: 0

Because of the space issue it seems like it may be more related to #79, but I am unsure of that.

Running 'find /etc /usr/local -size +1G -exec ls -lh {} ;' reveals that my apache logs are in /usr/local/... (we compiled apache by hand and this is the server root) and one of them (/usr/local/httpd/httpd-2.2.17/logs/access_log.00 is 1.3 GB.

Added the path as '/usr/local/httpd/httpd-2./logs/' to ~/.blueprintignore

Re-running blueprint create still results in the same error.

time blueprint create production-test-as-root
# [blueprint] parsing blueprintignore(5) rules
# [blueprint] searching for configuration files
# [blueprint] searching for APT packages
# [blueprint] searching for software built from source
# [blueprint] searching for Python packages
# [blueprint] searching for npm packages
# [blueprint] searching for PEAR/PECL packages
# [blueprint] searching for Yum packages
# [blueprint] searching for Ruby gems
# [blueprint] searching for service dependencies
error: 53e2ed69bf50ac48f20c52a2fa579191ce1635d1.tar: does not exist and --remove not passed
fatal: Unable to process path 53e2ed69bf50ac48f20c52a2fa579191ce1635d1.tar
Traceback (most recent call last):
  File "/usr/local/bin/blueprint-create", line 74, in 
    b = blueprint.cli.create(options, args)
  File "/usr/lib/python2.6/site-packages/blueprint/cli.py", line 39, in create
    b.commit(options.message or '')
  File "/usr/lib/python2.6/site-packages/blueprint/__init__.py", line 411, in commit
    git.git('update-index', '--add', os.path.abspath(filename))
  File "/usr/lib/python2.6/site-packages/blueprint/git.py", line 83, in git
    raise GitError(p.returncode)
blueprint.git.GitError: 128

real    15m30.596s
user    2m36.622s
sys     0m45.616s

Moved the blueprintignore file from /root/.blueprintignore to /etc/blueprintignore results in the same error, but it doesn't look like it re-read the file, probably because it has the same ctime, though it actually takes a minute longer to run.

time blueprint create production-test-as-root
# [blueprint] using cached blueprintignore(5) rules
# [blueprint] searching for configuration files
# [blueprint] searching for APT packages
# [blueprint] searching for software built from source
# [blueprint] searching for Python packages
# [blueprint] searching for npm packages
# [blueprint] searching for PEAR/PECL packages
# [blueprint] searching for Yum packages
# [blueprint] searching for Ruby gems
# [blueprint] searching for service dependencies
error: 24a6544825471f2c51325a1821b500591bd7f7df.tar: does not exist and --remove not passed
fatal: Unable to process path 24a6544825471f2c51325a1821b500591bd7f7df.tar
Traceback (most recent call last):
  File "/usr/local/bin/blueprint-create", line 74, in 
    b = blueprint.cli.create(options, args)
  File "/usr/lib/python2.6/site-packages/blueprint/cli.py", line 39, in create
    b.commit(options.message or '')
  File "/usr/lib/python2.6/site-packages/blueprint/__init__.py", line 411, in commit
    git.git('update-index', '--add', os.path.abspath(filename))
  File "/usr/lib/python2.6/site-packages/blueprint/git.py", line 83, in git
    raise GitError(p.returncode)
blueprint.git.GitError: 128

real    16m23.520s
user    2m37.678s
sys     0m46.271s

Perhaps the path was not specified correctly? Changing the blueprintignore path to /usr/local/httpd/httpd-2.2.17/logs/* results in the same error, which I won't post again.

Maybe it has something to do with the fact that I've moved the tmp.tar creation place? Though it seems that it would get moved to the right place before the git add. Do I just need to repartition and add more space to /tmp for this to actually work? Or is there some way to find out what in the world is chewing up so much disk space and causing git to fail?

[StephenBrown2]

Oh, also, my env is:

HOSTNAME=mybox
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=500
USER=root
LS_COLORS=
MAIL=/var/spool/mail/root
PATH=/usr/local/git/git-1.7.4.2/bin:/objs/bzr/bzr-2.3b1:/usr/local/drush:/usr/local/php/php-5.2.17/bin:/usr/local/httpd/httpd-2.2.17/bin:/usr/local/ncurses/ncurses-5.7/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/java/jre1.6.0_14/bin:/root/bin
INPUTRC=/etc/inputrc
PWD=/root
JAVA_HOME=/usr/java/jre1.6.0_14/bin/java
LANG=en_US.UTF-8
PS1=\[\e[1;31m\][\u@\h \w]\$\[\e[0m\]
SHLVL=1
HOME=/root
LOGNAME=root
LESSOPEN=|/usr/bin/lesspipe.sh %s
G_BROKEN_FILENAMES=1
_=/bin/env
OLDPWD=/usr/lib/python2.6/site-packages/blueprint/backend

and I get to root via 'sudo su -'

[xEtherealx]

I'm running out of memory in my /tmp folder as well (10Gb). Looking to set a custom temp directory but I've found nothing. Workaround for me is to ignore things that take a large amount of space (virtual machines residing in /usr/local/share).