Skip to content

Latest commit

 

History

History
4334 lines (3211 loc) · 165 KB

HISTORY.rst

File metadata and controls

4334 lines (3211 loc) · 165 KB
Raw

Historical ZODB Changelog

Contents

3.10.5 (2011-11-19)

Bugs Fixed

  • Conflict resolution failed when state included cross-database persistent references with classes that couldn't be imported.

3.10.4 (2011-11-17)

Bugs Fixed

  • Conflict resolution failed when state included persistent references with classes that couldn't be imported.

3.10.3 (2011-04-12)

Bugs Fixed

  • "activity monitor not updated for subconnections when connection returned to pool"

    https://bugs.launchpad.net/zodb/+bug/737198

  • "Blob temp file get's removed before it should", https://bugs.launchpad.net/zodb/+bug/595378

    A way this to happen is that a transaction is aborted after the commit process has started. I don't know how this would happen in the wild.

    In 3.10.3, the ZEO tpc_abort call to the server is changed to be synchronous, which should address this case. Maybe there's another case.

Performance enhancements

  • Improved ZEO client cache implementation to make it less likely to evict objects that are being used.
  • Small (possibly negligable) reduction in CPU in ZEO storage servers to service object loads and in networking code.

3.10.2 (2011-02-12)

Bugs Fixed

  • 3.10 introduced an optimization to try to address BTree conflict errors arrising for basing BTree keys on object ids. The optimization caused object ids allocated in aborted transactions to be reused. Unfortunately, this optimzation led to some rather severe failures in some applications. The symptom is a conflict error in which one of the serials mentioned is zero. This optimization has been removed.

    See (for example): https://bugs.launchpad.net/zodb/+bug/665452

  • ZEO server transaction timeouts weren't logged as critical.

    https://bugs.launchpad.net/zodb/+bug/670986

3.10.1 (2010-10-27)

Bugs Fixed

  • When a transaction rolled back a savepoint after adding objects and subsequently added more objects and committed, an error could be raised "ValueError: A different object already has the same oid" causing the transaction to fail. Worse, this could leave a database in a state where subsequent transactions in the same process would fail.

    https://bugs.launchpad.net/zodb/+bug/665452

  • Unix domain sockets didn't work for ZEO (since the addition of IPv6 support). https://bugs.launchpad.net/zodb/+bug/663259

  • Removed a missfeature that can cause performance problems when using an external garbage collector with ZEO. When objects were deleted from a storage, invalidations were sent to clients. This makes no sense. It's wildly unlikely that the other connections/clients have copies of the garbage. In normal storage garbage collection, we don't send invalidations. There's no reason to send them when an external garbage collector is used.

  • ZEO client cache simulation misshandled invalidations causing incorrect statistics and errors.

3.10.0 (2010-10-08)

New Features

  • There are a number of performance enhancements for ZEO storage servers.

  • FileStorage indexes use a new format. They are saved and loaded much faster and take less space. Old indexes can still be read, but new indexes won't be readable by older versions of ZODB.

  • The API for undoing multiple transactions has changed. To undo multiple transactions in a single transaction, pass a list of transaction identifiers to a database's undoMultiple method. Calling a database's undo method multiple times in the same transaction now raises an exception.

  • The ZEO protocol for undo has changed. The only user-visible consequence of this is that when ZODB 3.10 ZEO servers won't support undo for older clients.

  • The storage API (IStorage) has been tightened. Now, storages should raise a StorageTransactionError when invalid transactions are passed to tpc_begin, tpc_vote, or tpc_finish.

  • ZEO clients (ClientStorage instances) now work in forked processes, including those created via multiprocessing.Process instances.

  • Broken objects now provide the IBroken interface.

  • As a convenience, you can now pass an integer port as an address to the ZEO ClientStorage constructor.

  • As a convenience, there's a new client function in the ZEO package for constructing a ClientStorage instance. It takes the same arguments as the ClientStorage constructor.

  • DemoStorages now accept constructor athuments, close_base_on_close and close_changes_on_close, to control whether underlying storages are closed when the DemoStorage is closed.

    https://bugs.launchpad.net/zodb/+bug/118512

  • Removed the dependency on zope.proxy.

  • Removed support for the _p_independent mini framework, which was made moot by the introduction of multi-version concurrency control several years ago.

  • Added support for the transaction retry convenience (transaction-manager attempts method) introduced in the transaction 1.1.0 release.

  • Enhanced the database opening conveniences:

    • You can now pass storage keyword arguments to ZODB.DB and ZODB.connection.
    • You can now pass None (rather than a storage or file name) to get a database with a mapping storage.

  • Databases now warn when committing very large records (> 16MB). This is to try to warn people of likely design mistakes. There is a new option (large_record_size/large-record-size) to control the record size at which the warning is issued.

  • Added support for wrapper storages that transform pickle data. Applications for this include compression and encryption. An example wrapper storage implementation, ZODB.tests.hexstorage, was included for testing.

    It is important that storage implementations not assume that storages contain pickles. Renamed IStorageDB to IStorageWrapper and expanded it to provide methods for transforming and untransforming data records. Storages implementations should use these methods to get pickle data from stored records.

  • Deprecated ZODB.interfaces.StorageStopIteration. Storage iterator implementations should just raise StopIteration, which means they can now be implemented as generators.

  • The filestorage packer configuration option noe accepts values of the form modname:expression, allowing the use of packer factories with options.

  • Added a new API that allows applications to make sure that current data are read. For example, with:

    self._p_jar.readCurrent(ob)

    A conflict error will be raised if the version of ob read by the transaction isn't current when the transaction is committed.

    Normally, ZODB only assures that objects read are consistent, but not necessarily up to date. Checking whether an object is up to date is important when information read from one object is used to update another.

    BTrees are an important case of reading one object to update another. Internal nodes are read to decide which leave notes are updated when a BTree is updated. BTrees now use this new API to make sure that internal nodes are up to date on updates.

  • When transactions are aborted, new object ids allocated during the transaction are saved and used in subsequent transactions. This can help in situations where object ids are used as BTree keys and the sequential allocation of object ids leads to conflict errors.

  • ZEO servers now support a server_status method for for getting information on the number of clients, lock requests and general statistics.

  • ZEO clients now support a client_label constructor argument and client-label configuration-file option to specify a label for a client in server logs. This makes it easier to identify specific clients corresponding to server log entries, especially when there are multiple clients originating from the same machine.

  • Improved ZEO server commit lock logging. Now, locking activity is logged at the debug level until the number of waiting lock requests gets above 3. Log at the critical level when the number of waiting lock requests gets above 9.

  • The file-storage backup script, repozo, will now create a backup index file if an output file name is given via the --output/-o option.

  • Added a '--kill-old-on-full' argument to the repozo backup options: if passed, remove any older full or incremental backup files from the repository after doing a full backup. (https://bugs.launchpad.net/zope2/+bug/143158)

  • The mkzeoinst script has been moved to a separate project:

    https://pypi.org/project/zope.mkzeoinstance/

    and is no-longer included with ZODB.

  • Removed untested unsupported dbmstorage fossile.

  • ZEO servers no longer log their pids in every log message. It's just not interesting. :)

Bugs fixed

  • When a pool timeout was specified for a database and old connections were removed due to timing out, an error occured due to a bug in the connection cleanup logic.

  • When multi-database connections were no longer used and cleaned up, their subconnections weren't cleaned up properly.

  • ZEO didn't work with IPv6 addrsses. Added IPv6 support contributed by Martin v. Loewis.

  • A file storage bug could cause ZEO clients to have incorrect information about current object revisions after reconnecting to a database server.

  • Updated the 'repozo --kill-old-on-full' option to remove any '.index' files corresponding to backups being removed.

  • ZEO extension methods failed when a client reconnected to a storage. (https://bugs.launchpad.net/zodb/+bug/143344)

  • Clarified the return Value for lastTransaction in the case when there aren't any transactions. Now a string of 8 nulls (aka "z64") is specified.

  • Setting _p_changed on a blob wo actually writing anything caused an error. (https://bugs.launchpad.net/zodb/+bug/440234)

  • The verbose mode of the fstest was broken. (https://bugs.launchpad.net/zodb/+bug/475996)

  • Object ids created in a savepoint that is rolled back wren't being reused. (https://bugs.launchpad.net/zodb/+bug/588389)

  • Database connections didn't invalidate cache entries when conflict errors were raised in response to checkCurrentSerialInTransaction errors. Normally, this shouldn't be a problem, since there should be pending invalidations for these oids which will cause the object to be invalidated. There have been issues with ZEO persistent cache management that have caused out of date data to remain in the cache. (It's possible that the last of these were addressed in the 3.10.0b5.) Invalidating read data when there is a conflict error provides some extra insurance.

  • The interface, ZODB.interfaces.IStorage was incorrect. The store method should never return a sequence of oid and serial pairs.

  • When a demo storage push method was used to create a new demo storage and the new storage was closed, the original was (incorrectly) closed.

  • There were numerous bugs in the ZEO cache tracing and analysis code. Cache simulation, while not perfect, seems to be much more accurate now than it was before.

    The ZEO cache trace statistics and simulation scripts have been given more descriptive names and moved to the ZEO scripts package.

  • BTree sets and tree sets didn't correctly check values passed to update or to constructors, causing Python to exit under certain circumstances.

  • Fixed bug in copying a BTrees.Length instance. (https://bugs.launchpad.net/zodb/+bug/516653)

  • Fixed a serious bug that caused cache failures when run with Python optimization turned on.

    https://bugs.launchpad.net/zodb/+bug/544305

  • When using using a ClientStorage in a Storage server, there was a threading bug that caused clients to get disconnected.

  • On Mac OS X, clients that connected and disconnected quickly could cause a ZEO server to stop accepting connections, due to a failure to catch errors in the initial part of the connection process.

    The failure to properly handle exceptions while accepting connections is potentially problematic on other platforms.

    Fixes: https://bugs.launchpad.net/zodb/+bug/135108

  • Object state management wasn't done correctly when classes implemented custom _p_deavtivate methods. (https://bugs.launchpad.net/zodb/+bug/185066)

3.9.7 (2010-09-28)

Bugs Fixed

  • Changes in way that garbage collection treats dictionaries in Python 2.7 broke the object/connection cache implementation. (https://bugs.launchpad.net/zodb/+bug/641481)

    Python 2.7 wasn't officially supported, but we were releasing binaries for it, so ...

  • Logrotation/repoening via a SIGUSR2 signal wasn't implemented. (https://bugs.launchpad.net/zodb/+bug/143600)

  • When using multi-databases, cache-management operations on a connection, cacheMinimize and cacheGC, weren't applied to subconnections.

3.9.6 (2010-09-21)

Bugs Fixed

  • Updating blobs in save points could cause spurious "invalidations out of order" errors. https://bugs.launchpad.net/zodb/+bug/509801

    (Thanks to Christian Zagrodnick for chasing this down.)

  • If a ZEO client process was restarted while invalidating a ZEO cache entry, the cache could be left in a stage when there is data marked current that should be invalidated, leading to persistent conflict errors.

  • Corrupted or invalid cache files prevented ZEO clients from starting. Now, bad cache files are moved aside.

  • Invalidations of object records in ZEO caches, where the invalidation transaction ids matched the cached transaction ids should have been ignored.

  • Shutting down a process while committing a transaction or processing invalidations from the server could cause ZEO persistent client caches to have invalid data. This, in turn caused stale data to remain in the cache until it was updated.

  • Conflict errors didn't invalidate ZEO cache entries.

  • When objects were added in savepoints and either the savepoint was rolled back (https://bugs.launchpad.net/zodb/+bug/143560) or the transaction was aborted (https://mail.zope.org/pipermail/zodb-dev/2010-June/013488.html) The objects' _p_oid and _p_jar variables weren't cleared, leading to surprizing errors.

  • Objects added in transactions that were later aborted could have _p_changed still set (https://bugs.launchpad.net/zodb/+bug/615758).

  • ZEO extension methods failed when a client reconnected to a storage. (https://bugs.launchpad.net/zodb/+bug/143344)

  • On Mac OS X, clients that connected and disconnected quickly could cause a ZEO server to stop accepting connections, due to a failure to catch errors in the initial part of the connection process.

    The failure to properly handle exceptions while accepting connections is potentially problematic on other platforms.

    Fixes: https://bugs.launchpad.net/zodb/+bug/135108

  • Passing keys or values outside the range of 32-bit ints on 64-bit platforms led to undetected overflow errors. Now these cases cause Type errors to be raised.

    https://bugs.launchpad.net/zodb/+bug/143237

  • BTree sets and tree sets didn't correctly check values passed to update or to constructors, causing Python to exit under certain circumstances.

  • The verbose mode of the fstest was broken. (https://bugs.launchpad.net/zodb/+bug/475996)

3.9.5 (2010-04-23)

Bugs Fixed

3.9.4 (2009-12-14)

Bugs Fixed

  • A ZEO threading bug could cause transactions to read inconsistent data. (This sometimes caused an AssertionError in Connection._setstate_noncurrent.)
  • DemoStorage.loadBefore sometimes returned invalid data which would trigger AssertionErrors in ZODB.Connection.
  • History support was broken when using stprages that work with ZODB 3.8 and 3.9.
  • zope.testing was an unnecessary non-testing dependency.
  • Internal ZEO errors were logged at the INFO level, rather than at the error level.
  • The FileStorage backup and restore script, repozo, gave a deprecation warning under Python 2.6.
  • C Header files weren't installed correctly.
  • The undo implementation was incorrect in ways that could cause subtle missbehaviors.

3.9.3 (2009-10-23)

Bugs Fixed

  • 2 BTree bugs, introduced by a bug fix in 3.9.0c2, sometimes caused deletion of keys to be improperly handled, resulting in data being available via iteraation but not item access.

3.9.2 (2009-10-13)

Bugs Fixed

  • ZEO manages a separate thread for client network IO. It created this thread on import, which caused problems for applications that implemented daemon behavior by forking. Now, the client thread isn't created until needed.
  • File-storage pack clean-up tasks that can take a long time unnecessarily blocked other activity.
  • In certain rare situations, ZEO client connections would hang during the initial connection setup.

3.9.1 (2009-10-01)

Bugs Fixed

  • Conflict errors committing blobs caused ZEO servers to stop committing transactions.

3.9.0 (2009-09-08)

New Features (in more or less reverse chronological order)

  • The Database class now has an xrefs keyword argument and a corresponding allow-implicit-cross-references configuration option. which default to true. When set to false, cross-database references are disallowed.

  • Added support for RelStorage.

  • As a convenience, the connection root method for returning the root object can now also be used as an object with attributes mapped to the root-object keys.

  • Databases have a new method, transaction, that can be used with the Python (2.5 and later) with statement:

    db = ZODB.DB(...)
with db.transaction() as conn:
     # ... do stuff with conn

    This uses a private transaction manager for the connection. If control exits the block without an error, the transaction is committed, otherwise, it is aborted.

  • Convenience functions ZODB.connection and ZEO.connection provide a convenient way to open a connection to a database. They open a database and return a connection to it. When the connection is closed, the database is closed as well.

  • The ZODB.config databaseFrom... methods now support multi-databases. If multiple zodb sections are used to define multiple databases, the databases are connected in a multi-database arrangement and the first of the defined databases is returned.

  • The zeopack script has gotten a number of improvements:

    • Simplified command-line interface. (The old interface is still supported, except that support for ZEO version 1 servers has been dropped.)
    • Multiple storages can be packed in sequence.
      • This simplifies pack scheduling on servers serving multiple databases.
      • All storages are packed to the same time.
    • You can now specify a time of day to pack to.
    • The script will now time out if it can't connect to s storage in 60 seconds.

  • The connection now estimates the object size based on its pickle size and informs the cache about size changes.

    The database got additional configurations options (cache-size-bytes and historical-cache-size-bytes) to limit the cache size based on the estimated total size of cached objects. The default values are 0 which has the interpretation "do not limit based on the total estimated size". There are corresponding methods to read and set the new configuration parameters.

  • Connections now have a public opened attribute that is true when the connection is open, and false otherwise. When true, it is the seconds since the epoch (time.time()) when the connection was opened. This is a renaming of the previous _opened private variable.

  • FileStorage now supports blobs directly.

  • You can now control whether FileStorages keep .old files when packing.

  • POSKeyErrors are no longer logged by ZEO servers, because they are really client errors.

  • A new storage interface, IExternalGC, to support external garbage collection, http://wiki.zope.org/ZODB/ExternalGC, has been defined and implemented for FileStorage and ClientStorage.

  • As a small convenience (mainly for tests), you can now specify initial data as a string argument to the Blob constructor.

  • ZEO Servers now provide an option, invalidation-age, that allows quick verification of ZEO clients have been disconnected for less than a given time even if the number of transactions the client hasn't seen exceeds the invalidation queue size. This is only recommended if the storage being served supports efficient iteration from a point near the end of the transaction history.

  • The FileStorage iterator now handles large files better. When iterating from a starting transaction near the end of the file, the iterator will scan backward from the end of the file to find the starting point. This enhancement makes it practical to take advantage of the new storage server invalidation-age option.

  • Previously, database connections were managed as a stack. This tended to cause the same connection(s) to be used over and over. For example, the most used connection would typically be the only connection used. In some rare situations, extra connections could be opened and end up on the top of the stack, causing extreme memory wastage. Now, when connections are placed on the stack, they sink below existing connections that have more active objects.

  • There is a new pool-timeout database configuration option to specify that connections unused after the given time interval should be garbage collection. This will provide a means of dealing with extra connections that are created in rare circumstances and that would consume an unreasonable amount of memory.

  • The Blob open method now supports a new mode, 'c', to open committed data for reading as an ordinary file, rather than as a blob file. The ordinary file may be used outside the current transaction and even after the blob's database connection has been closed.

  • ClientStorage now provides blob cache management. When using non-shared blob directories, you can set a target cache size and the cache will periodically be reduced try to keep it below the target size.

    The client blob directory layout has changed. If you have existing non-shared blob directories, you will have to remove them.

  • ZODB 3.9 ZEO clients can connect to ZODB 3.8 servers. ZODB ZEO clients from ZODB 3.2 on can connect to ZODB 3.9 servers.

  • When a ZEO cache is stale and would need verification, a ZEO.interfaces.StaleCache event is published (to zope.event). Applications may handle this event and take action such as exiting the application without verifying the cache or starting cold.

  • There's a new convenience function, ZEO.DB, for creating databases using ZEO Client Storages. Just call ZEO.DB with the same arguments you would otherwise pass to ZEO.ClientStorage.ClientStorage:

    import ZEO
db = ZEO.DB(('some_host', 8200))

  • Object saves are a little faster

  • When configuring storages in a storage server, the storage name now defaults to "1". In the overwhelmingly common case that a single storage, the name can now be omitted.

  • FileStorage now provides optional garbage collection. A 'gc' keyword option can be passed to the pack method. A false value prevents garbage collection.

  • The FileStorage constructor now provides a boolean pack_gc option, which defaults to True, to control whether garbage collection is performed when packing by default. This can be overridden with the gc option to the pack method.

    The ZConfig configuration for FileStorage now includes a pack-gc option, corresponding to the pack_gc constructor argument.

  • The FileStorage constructor now has a packer keyword argument that allows an alternative packer to be supplied.

    The ZConfig configuration for FileStorage now includes a packer option, corresponding to the packer constructor argument.

  • MappingStorage now supports multi-version concurrency control and iteration and provides a better storage implementation example.

  • DemoStorage has a number of new features:

    • The ability to use a separate storage, such as a file storage to store changes
    • Blob support
    • Multi-version concurrency control and iteration
    • Explicit support for demo-storage stacking via push and pop methods.

  • Wen calling ZODB.DB to create a database, you can now pass a file name, rather than a storage to use a file storage.

  • Added support for copying and recovery of blob storages:

    • Added a helper function, ZODB.blob.is_blob_record for testing whether a data record is for a blob. This can be used when iterating over a storage to detect blob records so that blob data can be copied.

      In the future, we may want to build this into a blob-aware iteration interface, so that records get blob file attributes automatically.

    • Added the IBlobStorageRestoreable interfaces for blob storages that support recovery via a restoreBlob method.

    • Updated ZODB.blob.BlobStorage to implement IBlobStorageRestoreable and to have a copyTransactionsFrom method that also copies blob data.

  • New ClientStorage configuration option drop_cache_rather_verify. If this option is true then the ZEO client cache is dropped instead of the long (unoptimized) verification. For large caches, setting this option can avoid effective down times in the order of hours when the connection to the ZEO server was interrupted for a longer time.

  • Cleaned-up the storage iteration API and provided an iterator implementation for ZEO.

  • Versions are no-longer supported.

  • Document conflict resolution (see ZODB/ConflictResolution.txt).

  • Support multi-database references in conflict resolution.

  • Make it possible to examine oid and (in some situations) database name of persistent object references during conflict resolution.

  • Moved the 'transaction' module out of ZODB. ZODB depends upon this module, but it must be installed separately.

  • ZODB installation now requires setuptools.

  • Added offset information to output of fstail script. Added test harness for this script.

  • Added support for read-only, historical connections based on datetimes or serials (TIDs). See src/ZODB/historical_connections.txt.

  • Removed the ThreadedAsync module.

  • Now depend on zc.lockfile

Bugs Fixed

  • CVE-2009-2701: Fixed a vulnerability in ZEO storage servers when blobs are available. Someone with write access to a ZEO server configured to support blobs could read any file on the system readable by the server process and remove any file removable by the server process.

  • BTrees (and TreeSets) kept references to internal keys. https://bugs.launchpad.net/zope3/+bug/294788

  • BTree Sets and TreeSets don't support the standard set add method. (Now either add or the original insert method can be used to add an object to a BTree-based set.)

  • The runzeo script didn't work without a configuration file. (https://bugs.launchpad.net/zodb/+bug/410571)

  • Officially deprecated PersistentDict (https://bugs.launchpad.net/zodb/+bug/400775)

  • Calling __setstate__ on a persistent object could under certain uncommon cause the process to crash. (https://bugs.launchpad.net/zodb/+bug/262158)

  • When committing transactions involving blobs to ClientStorages with non-shared blob directories, a failure could occur in tpc_finish if there was insufficient disk space to copy the blob file or if the file wasn't available. https://bugs.launchpad.net/zodb/+bug/224169

  • Savepoint blob data wasn't properly isolated. If multiple simultaneous savepoints in separate transactions modified the same blob, data from one savepoint would overwrite data for another.

  • Savepoint blob data wasn't cleaned up after a transaction abort. https://bugs.launchpad.net/zodb/+bug/323067

  • Opening a blob with modes 'r+' or 'a' would fail when the blob had no committed changes.

  • PersistentList's sort method did not allow passing of keyword parameters. Changed its sort parameter list to match that of its (Python 2.4+) UserList base class.

  • Certain ZEO server errors could cause a client to get into a state where it couldn't commit transactions. https://bugs.launchpad.net/zodb/+bug/374737

  • Fixed vulnerabilities in the ZEO network protocol that allow:

    • CVE-2009-0668 Arbitrary Python code execution in ZODB ZEO storage servers
    • CVE-2009-0669 Authentication bypass in ZODB ZEO storage servers

    The vulnerabilities only apply if you are using ZEO to share a database among multiple applications or application instances and if untrusted clients are able to connect to your ZEO servers.

  • Fixed the setup test command. It previously depended on private functions in zope.testing.testrunner that don't exist any more.

  • ZEO client threads were unnamed, making it hard to debug thread management.

  • ZEO protocol 2 support was broken. This caused very old clients to be unable to use new servers.

  • zeopack was less flexible than it was before. -h should default to local host.

  • The "lawn" layout was being selected by default if the root of the blob directory happened to contain a hidden file or directory such as ".svn". Now hidden files and directories are ignored when choosing the default layout.

  • BlobStorage was not compatible with MVCC storages because the wrappers were being removed by each database connection. Fixed.

  • Saving indexes for large file storages failed (with the error: RuntimeError: maximum recursion depth exceeded). This can cause a FileStorage to fail to start because it gets an error trying to save its index.

  • Sizes of new objects weren't added to the object cache size estimation, causing the object-cache size limiting feature to let the cache grow too large when many objects were added.

  • Deleted records weren't removed when packing file storages.

  • Fixed analyze.py and added test.

  • fixed Python 2.6 compatibility issue with ZEO/zeoserverlog.py

  • using hashlib.sha1 if available in order to avoid DeprecationWarning under Python 2.6

  • made runzeo -h work

  • The monitor server didn't correctly report the actual number of clients.

  • Packing could return spurious errors due to errors notifying disconnected clients of new database size statistics.

  • Undo sometimes failed for FileStorages configured to support blobs.

  • Starting ClientStorages sometimes failed with non-new but empty cache files.

  • The history method on ZEO clients failed.

  • Fix for bug #251037: Make packing of blob storages non-blocking.

  • Fix for bug #220856: Completed implementation of ZEO authentication.

  • Fix for bug #184057: Make initialisation of small ZEO client file cache sizes not fail.

  • Fix for bug #184054: MappingStorage used to raise a KeyError during load instead of a POSKeyError.

  • Fixed bug in Connection.TmpStore: load() would not defer to the backend storage for loading blobs.

  • Fix for bug #181712: Make ClientStorage update lastTransaction directly after connecting to a server, even when no cache verification is necessary.

  • Fixed bug in blob filesystem helper: the isSecure check was inverted.

  • Fixed bug in transaction buffer: a tuple was unpacked incorrectly in clear.

  • Bugfix the situation in which comparing persistent objects (for instance, as members in BTree set or keys of BTree) might cause data inconsistency during conflict resolution.

  • Fixed bug 153316: persistent and BTrees were using int for memory sizes which caused errors on x86_64 Intel Xeon machines (using 64-bit Linux).

  • Fixed small bug that the Connection.isReadOnly method didn't work after a savepoint.

  • Bug #98275: Made ZEO cache more tolerant when invalidating current versions of objects.

  • Fixed a serious bug that could cause client I/O to stop (hang). This was accompanied by a critical log message along the lines of: "RuntimeError: dictionary changed size during iteration".

  • Fixed bug #127182: Blobs were subclassable which was not desired.

  • Fixed bug #126007: tpc_abort had untested code path that was broken.

  • Fixed bug #129921: getSize() function in BlobStorage could not deal with garbage files

  • Fixed bug in which MVCC would not work for blobs.

  • Fixed bug in ClientCache that occurred with objects larger than the total cache size.

  • When an error occured attempting to lock a file and logging of said error was enabled.

  • FileStorages previously saved indexes after a certain number of writes. This was done during the last phase of two-phase commit, which made this critical phase more subject to errors than it should have been. Also, for large databases, saves were done so infrequently as to be useless. The feature was removed to reduce the chance for errors during the last phase of two-phase commit.

  • File storages previously kept an internal object id to transaction id mapping as an optimization. This mapping caused excessive memory usage and failures during the last phase of two-phase commit. This optimization has been removed.

  • Refactored handling of invalidations on ZEO clients to fix a possible ordering problem for invalidation messages.

  • On many systems, it was impossible to create more than 32K blobs. Added a new blob-directory layout to work around this limitation.

  • Fixed bug that could lead to memory errors due to the use of a Python dictionary for a mapping that can grow large.

  • Fixed bug #251037: Made packing of blob storages non-blocking.

  • Fixed a bug that could cause InvalidObjectReference errors for objects that were explicitly added to a database if the object was modified after a savepoint that added the object.

  • Fixed several bugs that caused ZEO cache corruption when connecting to servers. These bugs affected both persistent and non-persistent caches.

  • Improved the the ZEO client shutdown support to try to avoid spurious errors on exit, especially for scripts, such as zeopack.

  • Packing failed for databases containing cross-database references.

  • Cross-database references to databases with empty names weren't constructed properly.

  • The zeo client cache used an excessive amount of memory, causing applications with large caches to exhaust available memory.

  • Fixed a number of bugs in the handling of persistent ZEO caches:

    • Cache records are written in several steps. If a process exits after writing begins and before it is finishes, the cache will be corrupt on restart. The way records are written was changed to make cache record updates atomic.
    • There was no lock file to prevent opening a cache multiple times at once, which would lead to corruption. Persistent caches now use lock files, in the same way that file storages do.
    • A bug in the cache-opening logic led to cache failure in the unlikely event that a cache has no free blocks.

  • When using ZEO Client Storages, Errors occured when trying to store objects too big to fit in the ZEO cache file.

  • Fixed bug in blob filesystem helper: the isSecure check was inverted.

  • Fixed bug in transaction buffer: a tuple was unpacked incorrectly in clear.

  • Fixed bug in Connection.TmpStore: load() would not defer to the back-end storage for loading blobs.

  • Fixed bug #190884: Wrong reference to POSKeyError caused NameError.

  • Completed implementation of ZEO authentication. This fixes issue 220856.

What's new in ZODB 3.8.0

General

  • (unreleased) Fixed setup.py use of setuptools vs distutils, so .c and .h files are included in the bdist_egg.
  • The ZODB Storage APIs have been documented and cleaned up.
  • ZODB versions are now officially deprecated and support for them will be removed in ZODB 3.9. (They have been widely recognized as deprecated for quite a while.)
  • Changed the automatic garbage collection when opening a connection to only apply the garbage collections on those connections in the pool that are closed. (This fixed issue 113923.)

ZEO

  • (3.8a1) ZEO's strategoes for avoiding client cache verification were improved in the case that servers are restarted. Before, if transactions were committed after the restart, clients that were up to date or nearly up to date at the time of the restart and then connected had to verify their caches. Now, it is far more likely that a client that reconnects soon after a server restart won't have to verify its cache.
  • (3.8a1) Fixed a serious bug that could cause clients that disconnect from and reconnect to a server to get bad invalidation data if the server serves multiple storages with active writes.
  • (3.8a1) It is now theoretically possible to use a ClientStorage in a storage server. This might make it possible to offload read load from a storage server at the cost of increasing write latency. This should increase write throughput by offloading reads from the final storage server. This feature is somewhat experimental. It has tests, but hasn't been used in production.

Transactions

  • (3.8a1) Add a doom() and isDoomed() interface to the transaction module.

    First step towards the resolution of http://www.zope.org/Collectors/Zope3-dev/655

    A doomed transaction behaves exactly the same way as an active transaction but raises an error on any attempt to commit it, thus forcing an abort.

    Doom is useful in places where abort is unsafe and an exception cannot be raised. This occurs when the programmer wants the code following the doom to run but not commit. It is unsafe to abort in these circumstances as a following get() may implicitly open a new transaction.

    Any attempt to commit a doomed transaction will raise a DoomedTransaction exception.

  • (3.8a1) Clean up the ZODB imports in transaction.

    Clean up weird import dance with ZODB. This is unnecessary since the transaction module stopped being imported in ZODB/__init__.py in rev 39622.

  • (3.8a1) Support for subtransactions has been removed in favor of save points.

Blobs

  • (3.8b1) Updated the Blob implementation in a number of ways. Some of these are backward incompatible with 3.8a1:

    o The Blob class now lives in ZODB.blob

    o The blob openDetached method has been replaced by the committed method.

  • (3.8a1) Added new blob feature. See the ZODB/Blobs directory for documentation.

    ZODB now handles (reasonably) large binary objects efficiently. Useful to use from a few kilobytes to at least multiple hundred megabytes.

BTrees

  • (3.8a1) Added support for 64-bit integer BTrees as separate types.

    (For now, we're retaining compile-time support for making the regular integer BTrees 64-bit.)

  • (3.8a1) Normalize names in modules so that BTrees, Buckets, Sets, and TreeSets can all be accessed with those names in the modules (e.g., BTrees.IOBTree.BTree). This is in addition to the older names (e.g., BTrees.IOBTree.IOBTree). This allows easier drop-in replacement, which can especially be simplify code for packages that want to support both 32-bit and 64-bit BTrees.

  • (3.8a1) Describe the interfaces for each module and actually declare the interfaces for each.

  • (3.8a1) Fix module references so klass.__module__ points to the Python wrapper module, not the C extension.

  • (3.8a1) introduce module families, to group all 32-bit and all 64-bit modules.

What's new in ZODB3 3.7.0

Release date: 2007-04-20

Packaging

  • (3.7.0b3) ZODB is now packaged without it's dependencies

    ZODB no longer includes copies of dependencies such as ZConfig, zope.interface and so on. It now treats these as dependencies. If ZODB is installed with easy_install or zc.buildout, the dependencies will be installed automatically.

  • (3.7.0b3) ZODB is now a buildout

    ZODB checkouts are now built and tested using zc.buildout.

  • (3.7b4) Added logic to avoid spurious errors from the logging system on exit.

  • (3.7b2) Removed the "sync" mode for ClientStorage.

    Previously, a ClientStorage could be in either "sync" mode or "async" mode. Now there is just "async" mode. There is now a dedicicated asyncore main loop dedicated to ZEO clients.

    Applications no-longer need to run an asyncore main loop to cause client storages to run in async mode. Even if an application runs an asyncore main loop, it is independent of the loop used by client storages.

    This addresses a test failure on Mac OS X, http://www.zope.org/Collectors/Zope3-dev/650, that I believe was due to a bug in sync mode. Some asyncore-based code was being called from multiple threads that didn't expect to be.

    Converting to always-async mode revealed some bugs that weren't caught before because the tests ran in sync mode. These problems could explain some problems we've seen at times with clients taking a long time to reconnect after a disconnect.

    Added a partial heart beat to try to detect lost connections that aren't otherwise caught, http://mail.zope.org/pipermail/zodb-dev/2005-June/008951.html, by perioidically writing to all connections during periods of inactivity.

Connection management

  • (3.7a1) When more than pool_size connections have been closed, DB forgets the excess (over pool_size) connections closed first. Python's cyclic garbage collection can take "a long time" to reclaim them (and may in fact never reclaim them if application code keeps strong references to them), but such forgotten connections can never be opened again, so their caches are now cleared at the time DB forgets them. Most applications won't notice a difference, but applications that open many connections, and/or store many large objects in connection caches, and/or store limited resources (such as RDB connections) in connection caches may benefit.

BTrees

  • Support for 64-bit integer keys and values has been provided as a compile-time option for the "I" BTrees (e.g. IIBTree).

Documentation

  • (3.7a1) Thanks to Stephan Richter for converting many of the doctest files to ReST format. These are now chapters in the Zope 3 apidoc too.

IPersistent

  • (3.7a1) The documentation for _p_oid now specifies the concrete type of oids (in short, an oid is either None or a non-empty string).

Testing

  • (3.7b2) Fixed test-runner output truncation.

    A bug was fixed in the test runner that caused result summaries to be omitted when running on Windows.

Tools

  • (3.7a1) The changeover from zLOG to the logging module means that some tools need to perform minimal logging configuration themselves. Changed the zeoup script to do so and thus enable it to emit error messages.

BTrees

Connection

  • (3.7a1) An optimization for loading non-current data (MVCC) was inadvertently disabled in _setstate(); this has been repaired.

persistent

After Commit hooks

  • (3.7a1) Transaction objects have a new method, addAfterCommitHook(hook, *args, **kws). Hook functions registered with a transaction are called after the transaction commits or aborts. For example, one might want to launch non transactional or asynchrnonous code after a successful, or aborted, commit. See test_afterCommitHook() in transaction/tests/test_transaction.py for a tutorial doctest, and the ITransaction interface for details.

What's new in ZODB3 3.6.2?

Release date: 15-July-2006

DemoStorage

Following is combined news from internal releases (to support ongoing Zope2 / Zope3 development). These are the dates of the internal releases:

  • 3.6.1 27-Mar-2006
  • 3.6.0 05-Jan-2006
  • 3.6b6 01-Jan-2006
  • 3.6b5 18-Dec-2005
  • 3.6b4 04-Dec-2005
  • 3.6b3 06-Nov-2005
  • 3.6b2 25-Oct-2005
  • 3.6b1 24-Oct-2005
  • 3.6a4 07-Oct-2005
  • 3.6a3 07-Sep-2005
  • 3.6a2 06-Sep-2005
  • 3.6a1 04-Sep-2005

Removal of Features Deprecated in ZODB 3.4

(3.6b2) ZODB 3.6 no longer contains features officially deprecated in the ZODB 3.4 release. These include:

  • get_transaction(). Use transaction.get() instead. transaction.commit() is a shortcut spelling of transaction.get().commit(), and transaction.abort() of transaction.get().abort(). Note that importing ZODB no longer installs get_transaction as a name in Python's __builtin__ module either.
  • The begin() method of Transaction objects. Use the begin() method of a transaction manager instead. transaction.begin() is a shortcut spelling to call the default transaction manager's begin() method.
  • The dt argument to Connection.cacheMinimize().
  • The Connection.cacheFullSweep() method. Use cacheMinimize() instead.
  • The Connection.getTransaction() method. Pass a transaction manager to DB.open() instead.
  • The Connection.getLocalTransaction() method. Pass a transaction manager to DB.open() instead.
  • The cache_deactivate_after and version_cache_deactivate_after arguments to the DB constructor.
  • The temporary, force, and waitflag arguments to DB.open(). DB.open() no longer blocks (there's no longer a fixed limit on the number of open connections).
  • The transaction and txn_mgr``arguments to ``DB.open(). Use the transaction_manager argument instead.
  • The getCacheDeactivateAfter, setCacheDeactivateAfter, getVersionCacheDeactivateAfter and setVersionCacheDeactivateAfter methods of DB.

Persistent

  • (3.6.1) Suppressed warnings about signedness of characters when compiling under GCC 4.0.x. See http://www.zope.org/Collectors/Zope/2027.
  • (3.6a4) ZODB 3.6 introduces a change to the basic behavior of Persistent objects in a particular end case. Before ZODB 3.6, setting obj._p_changed to a true value when obj was a ghost was ignored: obj remained a ghost, and getting obj._p_changed continued to return None. Starting with ZODB 3.6, obj is activated instead (unghostified), and its state is changed from the ghost state to the changed state. The new behavior is less surprising and more robust.
  • (3.6b5) The documentation for _p_oid now specifies the concrete type of oids (in short, an oid is either None or a non-empty string).

Commit hooks

  • (3.6a1) The beforeCommitHook() method has been replaced by the new addBeforeCommitHook() method, with a more-robust signature. beforeCommitHook() is now deprecated, and will be removed in ZODB 3.8. Thanks to Julien Anguenot for contributing code and tests.

Connection management

  • (3.6b6) When more than pool_size connections have been closed, DB forgets the excess (over pool_size) connections closed first. Python's cyclic garbage collection can take "a long time" to reclaim them (and may in fact never reclaim them if application code keeps strong references to them), but such forgotten connections can never be opened again, so their caches are now cleared at the time DB forgets them. Most applications won't notice a difference, but applications that open many connections, and/or store many large objects in connection caches, and/or store limited resources (such as RDB connections) in connection caches may benefit.

ZEO

  • (3.6a4) Collector 1900. In some cases of pickle exceptions raised by low-level ZEO communication code, callers of marshal.encode() could attempt to catch an exception that didn't actually exist, leading to an erroneous AttributeError exception. Thanks to Tres Seaver for the diagnosis.

BaseStorage

  • (3.6a4) Nothing done by tpc_abort() should raise an exception. However, if something does (an error case), BaseStorage.tpc_abort() left the commit lock in the acquired state, causing any later attempt to commit changes hang.

Multidatabase

  • (3.6b1) The database_name for a database in a multidatabase collection can now be specified in a config file's <zodb> section, as the value of the optional new database_name key. The .databases attribute cannot be specified in a config file, but can be passed as the optional new databases argument to the open() method of a ZConfig factory for type ZODBDatabase. For backward compatibility, Zope 2.9 continues to allow using the name in its <zodb_db name> config section as the database name (note that <zodb_db> is defined by Zope, not by ZODB -- it's a Zope-specific extension of ZODB's <zodb> section).

PersistentMapping

  • (3.6.1) PersistentMapping was inadvertently pickling volatile attributes (http://www.zope.org/Collectors/Zope/2052).
  • (3.6b4) PersistentMapping makes changes by a pop() method call persistent now (http://www.zope.org/Collectors/Zope/2036).
  • (3.6a1) The PersistentMapping class has an __iter__() method now, so that objects of this type work well with Python's iteration protocol. For example, if x is a PersistentMapping (or Python dictionary, or BTree, or PersistentDict, ...), then for key in x: iterates over the keys of x, list(x) creates a list containing x's keys, iter(x) creates an iterator for x's keys, and so on.

Tools

  • (3.6b5) The changeover from zLOG to the logging module means that some tools need to perform minimal logging configuration themselves. Changed the zeoup script to do so and thus enable it to emit error messages.

BTrees

  • (3.6.1) Suppressed warnings about signedness of characters when compiling under GCC 4.0.x. See http://www.zope.org/Collectors/Zope/2027.
  • (3.6a1) BTrees and Buckets now implement the setdefault() and pop() methods. These are exactly like Python's dictionary methods of the same names, except that setdefault() requires both arguments (and Python is likely to change to require both arguments too -- defaulting the default argument to None has no viable use cases). Thanks to Ruslan Spivak for contributing code, tests, and documentation.
  • (3.6a1) Collector 1873. It wasn't possible to construct a BTree or Bucket from, or apply their update() methods to, a PersistentMapping or PersistentDict. This works now.

ZopeUndo

  • (3.6a4) Collector 1810. A previous bugfix (#1726) broke listing undoable transactions for users defined in a non-root acl_users folder. Zope logs a acl_users path together with a username (separated by a space) and this previous fix failed to take this into account.

Connection

  • (3.6b5) An optimization for loading non-current data (MVCC) was inadvertently disabled in _setstate(); this has been repaired.

Documentation

  • (3.6b3) Thanks to Stephan Richter for converting many of the doctest files to ReST format. These are now chapters in the Zope 3 apidoc too.
  • (3.6b4) Several misspellings of "occurred" were repaired.

Development

  • (3.6a1) The source code for the old ExtensionClass-based Persistence package moved, from ZODB to the Zope 2.9 development tree. ZODB 3.5 makes no use of Persistence, and, indeed, the Persistence package could not be compiled from a ZODB release, since some of the C header files needed appear only in Zope.
  • (3.6a3) Re-added the zeoctl module, for the same reasons mkzeoinst was re-added (see below).
  • (3.6a2) The mkzeoinst module was re-added to ZEO, because Zope3 has a script that expects to import it from there. ZODB's mkzeoinst script was rewritten to invoke the mkzeoinst module.

transact

  • (3.6b4) Collector 1959: The undocumented transact module no longer worked. It remains undocumented and untested, but thanks to Janko Hauser it's possible that it works again ;-).

What's new in ZODB3 3.5.1?

Release date: 26-Sep-2005

Following is combined news from internal releases (to support ongoing Zope3 development). These are the dates of the internal releases:

  • 3.5.1b2 07-Sep-2005
  • 3.5.1b1 06-Sep-2005

Build

  • (3.5.1b2) Re-added the zeoctl module, for the same reasons mkzeoinst was re-added (see below).
  • (3.5.1b1) The mkzeoinst module was re-added to ZEO, because Zope3 has a script that expects to import it from there. ZODB's mkzeoinst script was rewritten to invoke the mkzeoinst module.

ZopeUndo

  • (3.5.1) Collector 1810. A previous bugfix (#1726) broke listing undoable transactions for users defined in a non-root acl_users folder. Zope logs a acl_users path together with a username (separated by a space) and this previous fix failed to take this into account.

What's new in ZODB3 3.5.0?

Release date: 31-Aug-2005

Following is combined news from internal releases (to support ongoing Zope3 development). These are the dates of the internal releases:

  • 3.5a7 11-Aug-2005
  • 3.5a6 04-Aug-2005
  • 3.5a5 19-Jul-2005
  • 3.5a4 14-Jul-2005
  • 3.5a3 17-Jun-2005
  • 3.5a2 16-Jun-2005
  • 3.5a1 10-Jun-2005

Savepoints

  • (3.5.0) As for deprecated subtransaction commits, the intent was that making a savepoint would invoke incremental garbage collection on Connection memory caches, to try to reduce the number of objects in cache to the configured cache size. Due to an oversight, this didn't happen, and stopped happening for subtransaction commits too. Making a savepoint (or doing a subtransaction commit) does invoke cache gc now.
  • (3.5a3) When a savepoint is made, the states of objects modified so far are saved to a temporary storage (an instance of class TmpStore, although that's an internal implementation detail). That storage needs to implement the full storage API too, but was missing the loadBefore() method needed for MVCC to retrieve non-current revisions of objects. This could cause spurious errors if a transaction with a pending savepoint needed to fetch an older revision of some object.
  • (3.5a4) The ISavepoint interface docs said you could roll back to a given savepoint any number of times (until the transaction ends, or until you roll back to an earlier savepoint's state), but the implementation marked a savepoint as invalid after its first use. The implementation has been repaired, to match the docs.

ZEO client cache

  • (3.5a6) Two memory leaks in the ZEO client cache were repaired, a major one involving ZEO.cache.Entry objects, and a minor one involving empty lists.

Subtransactions are deprecated

  • (3.5a4) Subtransactions are deprecated, and will be removed in ZODB 3.7. Use savepoints instead. Savepoints are more powerful, and code using subtransactions does not mix well with code using savepoints (a subtransaction commit forces all current savepoints to become unusable, so code using subtransactions can hurt newer code trying to use savepoints). In general, a subtransaction commit done just to free memory can be changed from:

    transaction.commit(1)

    to:

    transaction.savepoint(True)

    That is, make a savepoint, and forget it. As shown, it's best to pass True for the optional optimistic argument in this case: because there's no possibility of asking for a rollback later, there's no need to insist that all data managers support rollback.

    In rarer cases, a subtransaction commit is followed later by a subtransaction abort. In that case, change the initial:

    transaction.commit(1)

    to:

    sp = transaction.savepoint()

    and in place of the subtransaction abort:

    transaction.abort(1)

    roll back the savepoint instead:

    sp.rollback()

  • (3.5a4) Internal uses of subtransactions (transaction commit() or abort() passing a true argument) were rewritten to use savepoints instead.

Multi-database

  • (3.5a1) Preliminary support for persistent cross-database references has been added. See ZODB/cross-database-references.txt for an introduction.

Tools

  • (3.5a6, 3.5a7) Collector #1847. The ZEO client cache tracing and simulation tools weren't updated to work with ZODB 3.3, and the introduction of MVCC required major reworking of the tracing and simulation code. These tools are in a working state again, although so far lightly tested on just a few applications. In doc/ZEO/, see the heavily revised trace.txt and cache.txt.
  • (3.5a5) Collector #1846: If an uncommitted transaction was found, fsrecover.py fell into an infinite loop.

Windows

  • (3.5a6) As developed in a long thread starting at http://mail.zope.org/pipermail/zope/2005-July/160433.html there appears to be a race bug in the Microsoft Windows socket implementation, rarely visible in ZEO when multiple processes try to create an "asyncore trigger" simultaneously. Windows-specific code in ZEO/zrpc/trigger.py changed to work around this bug when it occurs.

ThreadedAsync.LoopCallback

  • (3.5a5) This once again physically replaces Python's asyncore.loop function with its own loop function, because it turns out Zope relied on the seemingly unused LoopCallback.exit_status global, which was removed in the change described below. Python's asyncore.loop is again not invoked, so any breakpoints or debugging prints added to that are again "lost".
  • (3.5a4) This replaces Python's asyncore.loop function with its own, in order to get notified when loop() is first called. The signature of asyncore.loop changed in Python 2.4, but LoopCallback.loop's signature didn't change to match. The code here was repaired to be compatible with both old and new signatures, and also repaired to invoke Python's asyncore.loop() instead of replacing it entirely (so, for example, debugging prints added to Python's asyncore.loop won't be lost anymore).

FileStorage

  • (3.5a4) Collector #1830. In some error cases when reading a FileStorage index, the code referenced an undefined global.
  • (3.5a4) Collector #1822. The undoLog() and undoInfo() methods were changed in 3.4a9 to return the documented results. Alas, some pieces of (non-ZODB) code relied on the actual behavior. When the first and last arguments are both >= 0, these methods now treat them as if they were Python slice indices, including the first index but excluding the last index. This matches former behavior, although it contradicts older ZODB UML documentation. The documentation in ZODB.interfaces.IStorageUndoable was changed to match the new intent.
  • (3.5a2) The _readnext() method now returns the transaction size as the value of the "size" key. Thanks to Dieter Maurer for the patch, from http://mail.zope.org/pipermail/zodb-dev/2003-October/006157.html. "This is very valuable when you want to spot strange transaction sizes via Zope's 'Undo' tab".

BTrees

  • (3.5.a5) Collector 1843. When a non-integer was passed to a method like keys() of a Bucket or Set with integer keys, an internal error code was overlooked, leading to everything from "delayed errors" to segfaults. Such cases raise TypeError now, as intended.
  • (3.5a4) Collector 1831. The BTree minKey() and maxKey() methods gave a misleading message if no key satisfying the constraints existed in a non-empty tree.
  • (3.5a4) Collector 1829. Clarified that the minKey() and maxKey() methods raise an exception if no key exists satsifying the constraints.
  • (3.5a4) The ancient convert.py script was removed. It was intended to convert "old" BTrees to "new" BTrees, but the "old" BTree implementation was removed from ZODB years ago.

What's new in ZODB3 3.4.1?

Release date: 09-Aug-2005

Following are dates of internal releases (to support ongoing Zope 2 development) since ZODB 3.4's last public release:

  • 3.4.1b5 08-Aug-2005
  • 3.4.1b4 07-Aug-2005
  • 3.4.1b3 04-Aug-2005
  • 3.4.1b2 02-Aug-2005
  • 3.4.1b1 26-Jul-2005
  • 3.4.1a6 19-Jul-2005
  • 3.4.1a5 12-Jul-2005
  • 3.4.1a4 08-Jul-2005
  • 3.4.1a3 02-Jul-2005
  • 3.4.1a2 29-Jun-2005
  • 3.4.1a1 27-Jun-2005

Savepoints

  • (3.4.1a1) When a savepoint is made, the states of objects modified so far are saved to a temporary storage (an instance of class TmpStore, although that's an internal implementation detail). That storage needs to implement the full storage API too, but was missing the loadBefore() method needed for MVCC to retrieve non-current revisions of objects. This could cause spurious errors if a transaction with a pending savepoint needed to fetch an older revision of some object.
  • (3.4.1a5) The ISavepoint interface docs said you could roll back to a given savepoint any number of times (until the transaction ends, or until you roll back to an earlier savepoint's state), but the implementation marked a savepoint as invalid after its first use. The implementation has been repaired, to match the docs.
  • (3.4.1b4) Collector 1860: use an optimistic savepoint in ExportImport (there's no possiblity of rollback here, so no need to insist that the data manager support rollbacks).

ZEO client cache

  • (3.4.1b3) Two memory leaks in the ZEO client cache were repaired, a major one involving ZEO.cache.Entry objects, and a minor one involving empty lists.

Subtransactions

  • (3.4.1a5) Internal uses of subtransactions (transaction commit() or abort() passing a true argument) were rewritten to use savepoints instead. Application code is strongly encouraged to do this too: subtransactions are weaker, will be deprecated soon, and do not mix well with savepoints (when you do a subtransaction commit, all current savepoints are made unusable). In general, a subtransaction commit done just to free memory can be changed from:

    transaction.commit(1)

    to:

    transaction.savepoint(True)

    That is, make a savepoint, and forget it. As shown, it's best to pass True for the optional optimistic argument in this case: because there's no possibility of asking for a rollback later, there's no need to insist that all data managers support rollback.

    In rarer cases, a subtransaction commit is followed later by a subtransaction abort. In that case, change the initial:

    transaction.commit(1)

    to:

    sp = transaction.savepoint()

    and in place of the subtransaction abort:

    transaction.abort(1)

    roll back the savepoint instead:

    sp.rollback()

FileStorage

  • (3.4.1a3) Collector #1830. In some error cases when reading a FileStorage index, the code referenced an undefined global.
  • (3.4.1a2) Collector #1822. The undoLog() and undoInfo() methods were changed in 3.4a9 to return the documented results. Alas, some pieces of (non-ZODB) code relied on the actual behavior. When the first and last arguments are both >= 0, these methods now treat them as if they were Python slice indices, including the first index but excluding the last index. This matches former behavior, although it contradicts older ZODB UML documentation. The documentation in ZODB.interfaces.IStorageUndoable was changed to match the new intent.
  • (3.4.1a1) The UndoSearch._readnext() method now returns the transaction size as the value of the "size" key. Thanks to Dieter Maurer for the patch, from http://mail.zope.org/pipermail/zodb-dev/2003-October/006157.html. "This is very valuable when you want to spot strange transaction sizes via Zope's 'Undo' tab".

ThreadedAsync.LoopCallback

  • (3.4.1a6) This once again physically replaces Python's asyncore.loop function with its own loop function, because it turns out Zope relied on the seemingly unused LoopCallback.exit_status global, which was removed in the change described below. Python's asyncore.loop is again not invoked, so any breakpoints or debugging prints added to that are again "lost".
  • (3.4.1a1) This replaces Python's asyncore.loop function with its own, in order to get notified when loop() is first called. The signature of asyncore.loop changed in Python 2.4, but LoopCallback.loop's signature didn't change to match. The code here was repaired to be compatible with both old and new signatures, and also repaired to invoke Python's asyncore.loop() instead of replacing it entirely (so, for example, debugging prints added to Python's asyncore.loop won't be lost anymore).

Windows

  • (3.4.1b2) As developed in a long thread starting at http://mail.zope.org/pipermail/zope/2005-July/160433.html there appears to be a race bug in the Microsoft Windows socket implementation, rarely visible in ZEO when multiple processes try to create an "asyncore trigger" simultaneously. Windows-specific code in ZEO/zrpc/trigger.py changed to work around this bug when it occurs.

Tools

  • (3.4.1b1 thru 3.4.1b5) Collector #1847. The ZEO client cache tracing and simulation tools weren't updated to work with ZODB 3.3, and the introduction of MVCC required major reworking of the tracing and simulation code. These tools are in a working state again, although so far lightly tested on just a few applications. In doc/ZEO/, see the heavily revised trace.txt and cache.txt.
  • (3.4.1a6) Collector #1846: If an uncommitted transaction was found, fsrecover.py fell into an infinite loop.

DemoStorage

  • (3.4.1a1) The implementation of undoLog() was wrong in several ways; repaired.

BTrees

  • (3.4.1a6) Collector 1843. When a non-integer was passed to a method like keys() of a Bucket or Set with integer keys, an internal error code was overlooked, leading to everything from "delayed errors" to segfaults. Such cases raise TypeError now, as intended.
  • (3.4.1a4) Collector 1831. The BTree minKey() and maxKey() methods gave a misleading message if no key satisfying the constraints existed in a non-empty tree.
  • (3.4.1a3) Collector 1829. Clarified that the minKey() and maxKey() methods raise an exception if no key exists satsifying the constraints.

What's new in ZODB3 3.4?

Release date: 09-Jun-2005

Following is combined news from the "internal releases" (to support ongoing Zope 2.8 and Zope3 development) since the last public ZODB 3.4 release. These are the dates of the internal releases:

  • 3.4c2 06-Jun-2005
  • 3.4c1 03-Jun-2005
  • 3.4b3 27-May-2005
  • 3.4b2 26-May-2005

Connection, DB

  • (3.4b3) .transaction_manager is now a public attribute of IDataManager, and is the instance of ITransactionManager used by the data manager as its transaction manager. There was previously no way to ask a data manager which transaction manager it was using. It's intended that transaction_manager be treated as read-only.
  • (3.4b3) For sanity, the txn_mgr argument to DB.open(), Connection.__init__(), and Connection._setDB() has been renamed to transaction_manager. txn_mgr is still accepted, but is deprecated and will be removed in ZODB 3.6. Any code that was using the private ._txn_mgr attribute of Connection will break immediately.

Development

  • (3.4b2) ZODB's test.py is now a small driver for the shared zope.testing.testrunner. See the latter's documentation for command-line arguments.

Error reporting

  • (3.4c1) In the unlikely event that referencesf() reports an unpickling error (for example, a corrupt database can cause this), the message it produces no longer contains unprintable characters.

Tests

  • (3.4c2) checkCrossDBInvalidations suffered spurious failures too often on slow and/or busy machines. The test is willing to wait longer for success now.

What's new in ZODB3 3.4b1?

Release date: 19-May-2005

What follows is combined news from the "internal releases" (to support ongoing Zope 2.8 and Zope3 development) since the last public ZODB 3.4 release. These are the dates of the internal releases:

  • 3.4b1 19-May-2005
  • 3.4a9 12-May-2005
  • 3.4a8 09-May-2005
  • 3.4a7 06-May-2005
  • 3.4a6 05-May-2005
  • 3.4a5 25-Apr-2005
  • 3.4a4 23-Apr-2005
  • 3.4a3 13-Apr-2005
  • 3.4a2 03-Apr-2005

transaction

  • (3.4a7) If the first activity seen by a new ThreadTransactionManager was an explicit begin() call, then synchronizers registered after that (but still during the first transaction) were not communicated to the transaction object. As a result, the afterCompletion() methods of registered synchronizers weren't called when the first transaction ended.

  • (3.4a6) Doing a subtransaction commit erroneously processed invalidations, which could lead to an inconsistent view of the database. For example, let T be the transaction of which the subtransaction commit was a part. If T read a persistent object O's state before the subtransaction commit, did not commit new state of its own for O during its subtransaction commit, and O was modified before the subtransaction commit by a different transaction, then the subtransaction commit processed an invalidation for O, and the state T read for O originally was discarded in T. If T went on to access O again, it saw the newly committed (by a different transaction) state for O:

    o_attr = O.some_attribute
get_transaction().commit(True)
assert o_attr == O.some_attribute

    could fail, and despite that T never modifed O.

  • (3.4a4) Transactions now support savepoints. Savepoints allow changes to be periodically checkpointed within a transaction. You can then rollback to a previously created savepoint. See transaction/savepoint.txt.

  • (3.4a6) A getBeforeCommitHooks() method was added. It returns an iterable producing the registered beforeCommit hooks.

  • (3.4a6) The ISynchronizer interface has a new newTransaction() method. This is invoked whenever a transaction manager's begin() method is called. (Note that a transaction object's (as opposed to a transaction manager's) begin() method is deprecated, and newTransaction() is not called when using the deprecated method.)

  • (3.4a6) Relatedly, Connection implements ISynchronizer, and Connection's afterCompletion() and newTransaction() methods now call sync() on the underlying storage (if the underlying storage has such a method), in addition to processing invalidations. The practical implication is that storage synchronization will be done automatically now, whenever a transaction is explicitly started, and after top-level transaction commit or abort. As a result, Connection.sync() should virtually never be needed anymore, and will eventually be deprecated.

  • (3.4a3) Transaction objects have a new method, beforeCommitHook(hook, *args, **kws). Hook functions registered with a transaction are called at the start of a top-level commit, before any of the work is begun, so a hook function can perform any database operations it likes. See test_beforeCommitHook() in transaction/tests/test_transaction.py for a tutorial doctest, and the ITransaction interface for details. Thanks to Florent Guillaume for contributing code and tests.

  • (3.4a3) Clarifications were made to transaction interfaces.

Support for ZODB4 savepoint-aware data managers has been dropped

  • (3.4a4) In adding savepoint support, we dropped the attempted support for ZODB4 data managers that support savepoints. We don't think that this will affect anyone.

ZEO

  • (3.4a4) The ZODB and ZEO version numbers are now the same. Concretely:

    import ZODB, ZEO
assert ZODB.__version__ == ZEO.version

    no longer fails. If interested, see the README file for details about earlier version numbering schemes.

  • (3.4b1) ZConfig version 2.3 adds new socket address types, for smoother default behavior across platforms. The hostname portion of socket-binding-address defaults to an empty string, which acts like INADDR_ANY on Windows and Linux (bind to any interface). The hostname portion of socket-connection-address defaults to "127.0.0.1" (aka "localhost"). In config files, the types of zeo section keys address and monitor-address changed to socket-binding-address, and the type of the zeoclient section key server changed to socket-connection-address.

  • (3.4a4) The default logging setup in runzeo.py was broken. It was changed so that running runzeo.py from a command line now, and without using a config file, prints output to the console much as ZODB 3.2 did.

ZEO on Windows

Thanks to Mark Hammond for these runzeo.py enhancements on Windows:

  • (3.4b1) Collector 1788: Repair one of the new features below.
  • (3.4a4) A pid file (containing the process id as a decimal string) is created now for a ZEO server started via runzeo.py. External programs can read the pid from this file and derive a "signal name" used in a new signal-emulation scheme for Windows. This is only necessary on Windows, but the pid file is created on all platforms that implement os.getpid(), as long as the pid-filename option is set, or environment variable INSTANCE_HOME is defined. The pid-filename option can be set in a ZEO config file, or passed as the new --pid-file argument to runzeo.py.
  • (3.4a4) If available, runzeo.py now uses Zope's new 'Signal' mechanism for Windows, to implement clean shutdown and log rotation handlers for Windows. Note that the Python in use on the ZEO server must also have the Python Win32 extensions installed for this to be useful.

Tools

  • (3.4a4) fsdump.py now displays the size (in bytes) of data records. This actually went in several months go, but wasn't noted here at the time. Thanks to Dmitry Vasiliev for contributing code and tests.

FileStorage

  • (3.4a9) The undoLog() and undoInfo() methods almost always returned a wrong number of results, one too many if last < 0 (the default is such a case), or one too few if last >= 0. These have been repaired, new tests were added, and these methods are now documented in ZODB.interfaces.IStorageUndoable.
  • (3.4a2) A pdb.set_trace() call was mistakenly left in method FileStorage.modifiedInVersion().

ZConfig

  • (3.4b1) The "standalone" release of ZODB now includes ZConfig version 2.3.

DemoStorage

  • (3.4a4) Appropriate implementations of the storage API's registerDB() and new_oid() methods were added, delegating to the base storage. This was needed to support wrapping a ZEO client storage as a DemoStorage base storage, as some new Zope tests want to do.

BaseStorage

  • (3.4a4) new_oid()'s undocumented last= argument was removed. It was used only for internal recursion, and injured code sanity elsewhere because not all storages included it in their new_oid()'s signature. Straightening this out required adding last= everywhere, or removing it everywhere. Since recursion isn't actually needed, and there was no other use for last=, removing it everywhere was the obvious choice.

Tests

  • (3.4a3) The various flavors of the check2ZODBThreads and check7ZODBThreads tests are much less likely to suffer sproadic failures now.
  • (3.4a2) The test checkOldStyleRoot failed in Zope3, because of an obscure dependence on the Persistence package (which Zope3 doesn't use).

ZApplication

  • (3.4a8) The file ZApplication.py was moved, from ZODB to Zope(2). ZODB and Zope3 don't use it, but Zope2 does.
  • (3.4a7) The __call__ method didn't work if a non-None connection string argument was passed. Thanks to Stefan Holek for noticing.

What's new in ZODB3 3.4a1?

Release date: 01-Apr-2005

transaction

  • get_transaction() is officially deprecated now, and will be removed in ZODB 3.6. Use the transaction package instead. For example, instead of:

    import ZODB
...
get_transaction().commit()

    do:

    import transaction
...
transaction.commit()

DB

  • There is no longer a hard limit on the number of connections that DB.open() will create. In other words, DB.open() never blocks anymore waiting for an earlier connection to close, and DB.open() always returns a connection now (while it wasn't documented, it was possible for DB.open() to return None before).

    pool_size continues to default to 7, but its meaning has changed: if more than pool_size connections are obtained from DB.open() and not closed, a warning is logged; if more than twice pool_size, a critical problem is logged. pool_size should be set to the maximum number of connections from the DB instance you expect to have open simultaneously.

    In addition, if a connection obtained from DB.open() becomes unreachable without having been explicitly closed, when Python's garbage collection reclaims that connection it no longer counts against the pool_size thresholds for logging messages.

    The following optional arguments to DB.open() are deprecated: transaction, waitflag, force and temporary. If one is specified, its value is ignored, and DeprecationWarning is raised. In ZODB 3.6, these optional arguments will be removed.

  • Lightweight support for "multi-databases" is implemented. These are collections of named DB objects and associated open Connections, such that the Connection for any DB in the collection can be obtained from a Connection from any other DB in the collection. See the new test file ZODB/tests/multidb.txt for a tutorial doctest. Thanks to Christian Theune for his work on this during the PyCon 2005 ZODB sprint.

ZEO compatibility

There are severe restrictions on using ZEO servers and clients at or after ZODB 3.3 with ZEO servers and clients from ZODB versions before 3.3. See the reworked Compatibility section in README.txt for details. If possible, it will be easiest to move clients and servers to 3.3+ simultaneously. With care, it's possible to use a 3.3+ ZEO server with pre-3.3 ZEO clients, but not possible to use a pre-3.3 ZEO server with 3.3+ ZEO clients.

BTrees

  • A new family of BTree types, in the IFBTree module, map signed integers (32 bits) to C floats (also 32 bits). The intended use is to help construct search indices, where, e.g., integer word or document identifiers map to scores of some kind. This is easier than trying to work with scaled integer scores in an IIBTree, and Zope3 has moved to IFBTrees for these purposes in its search code.

FileStorage

  • Addded a record iteration protocol to FileStorage. You can use the record iterator to iterate over all current revisions of data pickles in the storage.

    In order to support calling via ZEO, we don't implement this as an actual iterator. An example of using the record iterator protocol is as follows:

    storage = FileStorage('anexisting.fs')
next_oid = None
while True:
    oid, tid, data, next_oid = storage.record_iternext(next_oid)
    # do something with oid, tid and data
    if next_oid is None:
        break

    The behavior of the iteration protocol is now to iterate over all current records in the database in ascending oid order, although this is not a promise to do so in the future.

Tools

New tool fsoids.py, for heavy debugging of FileStorages; shows all uses of specified oids in the entire database (e.g., suppose oid 0x345620 is missing -- did it ever exist? if so, when? who referenced it? when was the last transaction that modified an object that referenced it? which objects did it reference? what kind of object was it?). ZODB/test/testfsoids.py is a tutorial doctest.

fsIndex

Efficient, general implementations of minKey() and maxKey() methods were added. fsIndex is a special hybrid kind of BTree used to implement FileStorage indices. Thanks to Chris McDonough for code and tests.

What's new in ZODB3 3.3.1?

Release date: DD-MMM-2005

Tests

The various flavors of the check2ZODBThreads and check7ZODBThreads tests are much less likely to suffer sproadic failures now.

What's new in ZODB3 3.3.1c1?

Release date: 01-Apr-2005

BTrees

Collector #1734: BTrees conflict resolution leads to index inconsistencies.

Silent data loss could occur due to BTree conflict resolution when one transaction T1 added a new key to a BTree containing at least three buckets, and a concurrent transaction T2 deleted all keys in the bucket to which the new key was added. Conflict resolution then created a bucket containing the newly added key, but the bucket remained isolated, disconnected from the BTree. In other words, the committed BTree didn't contain the new key added by T1. Conflict resolution doesn't have enough information to repair this, so ConflictError is now raised in such cases.

ZEO

Repaired subtle race conditions in establishing ZEO connections, both client- and server-side. These account for intermittent cases where ZEO failed to make a connection (or reconnection), accompanied by a log message showing an error caught in asyncore and having a traceback ending with:

UnpicklingError: invalid load key, 'Z'.

or:

ZRPCError: bad handshake '(K\x00K\x00U\x0fgetAuthProtocol)t.'

or:

error: (9, 'Bad file descriptor')

or an AttributeError.

These were exacerbated when running the test suite, because of an unintended busy loop in the test scaffolding, which could starve the thread trying to make a connection. The ZEO reconnection tests may run much faster now, depending on platform, and should suffer far fewer (if any) intermittent "timed out waiting for storage to connect" failures.

ZEO protocol and compatibility

ZODB 3.3 introduced multiversion concurrency control (MVCC), which required changes to the ZEO protocol. The first 3.3 release should have increased the internal ZEO protocol version number (used by ZEO protocol negotiation when a client connects), but neglected to. This has been repaired.

Compatibility between pre-3.3 and post-3.3 ZEO clients and servers remains very limited. See the newly updated Compatibility section in README.txt for details.

FileStorage

  • The .store() and .restore() methods didn't update the storage's belief about the largest oid in use when passed an oid larger than the largest oid the storage already knew about. Because .restore() in particular is used by copyTransactionsFrom(), and by the first stage of ZRS recovery, a large database could be created that believed the only oid in use was oid 0 (the special oid reserved for the root object). In rare cases, it could go on from there assigning duplicate oids to new objects, starting over from oid 1 again. This has been repaired. A new set_max_oid() method was added to the BaseStorage class so that derived storages can update the largest oid in use in a threadsafe way.

  • A FileStorage's index file tried to maintain the index's largest oid as a separate piece of data, incrementally updated over the storage's lifetime. This scheme was more complicated than necessary, so was also more brittle and slower than necessary. It indirectly participated in a rare but critical bug: when a FileStorage was created via copyTransactionsFrom(), the "maximum oid" saved in the index file was always 0. Use that FileStorage, and it could then create "new" oids starting over at 0 again, despite that those oids were already in use by old objects in the database. Packing a FileStorage has no reason to try to update the maximum oid in the index file either, so this kind of damage could (and did) persist even across packing.

    The index file's maximum-oid data is ignored now, but is still written out so that .index files can be read by older versions of ZODB. Finding the true maximum oid is done now by exploiting that the main index is really a kind of BTree (long ago, this wasn't true), and finding the largest key in a BTree is inexpensive.

  • A FileStorage's index file could be updated on disk even if the storage was opened in read-only mode. That bug has been repaired.

  • An efficient maxKey() implementation was added to class fsIndex.

Pickle (in-memory Connection) Cache

You probably never saw this exception:

ValueError: Can not re-register object under a different oid

It's been changed to say what it meant:

ValueError: A different object already has the same oid

This happens if an attempt is made to add distinct objects to the cache that have the same oid (object identifier). ZODB should never do this, but it's possible for application code to force such an attempt.

PersistentMapping and PersistentList

Backward compatibility code has been added so that the sanest of the ZODB 3.2 dotted paths for PersistentMapping and PersistentList resolve. These are still preferred:

  • from persistent.list import PersistentList
  • from persistent.mapping import PersistentMapping

but these work again too:

  • from ZODB.PersistentList import PersistentList
  • from ZODB.PersistentMapping import PersistentMapping

BTrees

The BTrees interface file neglected to document the optional excludemin and excludemax arguments to the keys(), values() and items() methods. Appropriate changes were merged in from the ZODB4 BTrees interface file.

Tools

  • mkzeoinst.py's default port number changed from to 9999 to 8100, to match the example in Zope's zope.conf.

fsIndex

An efficient maxKey() method was implemented for the fsIndex class. This makes it possible to determine the largest oid in a FileStorage index efficiently, directly, and reliably, replacing a more delicate scheme that tried to keep track of this by saving an oid high water mark in the index file and incrementally updating it.

What's new in ZODB3 3.3.1a1?

Release date: 11-Jan-2005

ZEO client cache

  • Collector 1536: The cache-size configuration option for ZEO clients was being ignored. Worse, the client cache size was only one megabyte, much smaller than the advertised default of 20MB. Note that the default is carried over from a time when gigabyte disks were expensive and rare; 20MB is also too small on most modern machines.
  • Fixed a nasty bug in cache verification. A persistent ZEO cache uses a disk file, and, when active, has some in-memory data structures too to speed operation. Invalidations processed as part of startup cache verification were reflected in the in-memory data structures, but not correctly in the disk file. So if an object revision was invalidated as part of verification, the object wasn't loaded again before the connection was closed, and the object revision remained in the cache file until the connection was closed, then the next time the cache file was opened it could believe that the stale object revision in the file was actually current.
  • Fixed a bug wherein an object removed from the client cache didn't properly mark the file slice it occupied as being available for reuse.

ZEO

Collector 1503: excessive logging. It was possible for a ZEO client to log "waiting for cache verification to finish" messages at a very high rate, producing gigabytes of such messages in short order. ClientStorage._wait_sync() was changed to log no more than one such message per 5 minutes.

persistent

Collector #1350: ZODB has a default one-thread-per-connection model, and two threads should never do operations on a single connection simultaneously. However, ZODB can't detect violations, and this happened in an early stage of Zope 2.8 development. The low-level ghostify() and unghostify() routines in cPerisistence.c were changed to give some help in detecting this when it happens. In a debug build, both abort the process if thread interference is detected. This is extreme, but impossible to overlook. In a release build, unghostify() raises SystemError if thread damage is detected; ghostify() ignores the problem in a release build (ghostify() is supposed to be so simple that it "can't fail").

ConflictError

New in 3.3, a ConflictError exception may attempt to insert the path to the object's class in its message. However, a ZEO server may not have access to application class implementations, and then the attempt by the server to raise ConflictError could raise ImportError instead while trying to determine the object's class path. This was confusing. The code has been changed to obtain the class path from the object's pickle, without trying to import application modules or classes.

FileStorage

Collector 1581: When an attempt to pack a corrupted Data.fs file was made, it was possible for the pack routine to die with a reference to an undefined global while it was trying to raise CorruptedError. It raises CorruptedError, as it always intended, in these cases now.

Install

The C header file ring.h is now installed.

Tools

  • BTrees.check.display() now displays the oids (if any) of the BTree's or TreeSet's constituent objects.

What's new in ZODB3 3.3?

Release date: 06-Oct-2004

ZEO

The encoding of RPC calls between server and client was being done with protocol 0 ("text mode") pickles, which could require sending four times as many bytes as necessary. Protocol 1 pickles are used now. Thanks to Andreas Jung for the diagnosis and cure.

ZODB/component.xml

cache-size parameters were changed from type integer to type byte-size. This allows you to specify, for example, "cache-size 20MB" to get a 20 megabyte cache.

transaction

The deprecation warning for Transaction.begin() was changed to point to the caller, instead of to Transaction.begin() itself.

Connection

Restored Connection's private _opened attribute. This was still referenced by DB.connectionDebugInfo(), and Zope 2 calls the latter.

FileStorage

Collector #1517: History tab for ZPT does not work. FileStorage.history() was reading the user, description, and extension fields out of the object pickle, due to starting the read at a wrong location. Looked like cut-and-paste repetition of the same bug in FileStorage.FileIterator noted in the news for 3.3c1.

What's new in ZODB3 3.3 release candidate 1?

Release date: 14-Sep-2004

Connection

ZODB intends to raise ConnnectionStateError if an attempt is made to close a connection while modifications are pending (the connection is involved in a transaction that hasn't been abort()'ed or commit()'ed). It was missing the case where the only pending modifications were made in subtransactions. This has been fixed. If an attempt to close a connection with pending subtransactions is made now:

ConnnectionStateError: Cannot close a connection with a pending subtransaction

is raised.

transaction

  • Transactions have new, backward-incompatible behavior in one respect: if a Transaction.commit(), Transaction.commit(False), or Transaction.commit(True) raised an exception, prior behavior was that the transaction effectively aborted, and a new transaction began. A primary bad consequence was that, if in a sequence of subtransaction commits, one of the commits failed but the exception was suppressed, all changes up to and including the failing commit were lost, but later subtransaction commits in the sequence got no indication that something had gone wrong, nor did the final (top level) commit. This could easily lead to inconsistent data being committed, from the application's point of view.

    The new behavior is that a failing commit "sticks" until explicitly cleared. Now if an exception is raised by a commit() call (whether subtransaction or top level) on a Transaction object T:

    • Pending changes are aborted, exactly as they were for a failing commit before.
    • But T remains the current transaction object (if tm is T's transaction manger, tm.get() continues to return T).
    • All subsequent attempts to do T.commit(), T.join(), or T.register() raise the new TransactionFailedError exception. Note that if you try to modify a persistent object, that object's resource manager (usually a Connection object) will attempt to join() the failed transaction, and TransactionFailedError will be raised right away.

    So after a transaction or subtransaction commit fails, that must be explicitly cleared now, either by invoking abort() on the transaction object, or by invoking begin() on its transaction manager.

  • Some explanations of new transaction features in the 3.3a3 news were incorrect, and this news file has been retroactively edited to repair that. See news for 3.3a3 below.

  • If ReadConflictError was raised by an attempt to load an object with a _p_independent() method that returned false, attempting to commit the transaction failed to (re)raise ReadConflictError for that object. Note that ZODB intends to prevent committing a transaction in which a ReadConflictError occurred; this was an obscure case it missed.

  • Growing pains: ZODB 3.2 had a bug wherein Transaction.begin() didn't abort the current transaction if the only pending changes were in a subtransaction. In ZODB 3.3, it's intended that a transaction manager be used to effect begin() (instead of invoking Transaction.begin()), and calling begin() on a transaction manager didn't have this old bug. However, Transaction.begin() still exists in 3.3, and it had a worse bug: it never aborted the transaction (not even if changes were pending outside of subtransactions). Transaction.begin() has been changed to abort the transaction. Transaction.begin() is also deprecated. Don't use it. Use begin() on the relevant transaction manager instead. For example,

    >>> import transaction
>>> txn = transaction.begin()  # start a txn using the default TM

    if using the default ThreadTransactionManager (see news for 3.3a3 below). In 3.3, it's intended that a single Transaction object is used for exactly one transaction. So, unlike as in 3.2, when somtimes Transaction objects were reused across transactions, but sometimes weren't, when you do Transaction.begin() in 3.3 a brand new transaction object is created. That's why this use is deprecated. Code of the form:

    >>> txn = transaction.get()
>>> ...
>>> txn.begin()
>>> ...
>>> txn.commit()

    can't work as intended in 3.3, because txn is no longer the current Transaction object the instant txn.begin() returns.

BTrees

The BTrees __init__.py file is now just a comment. It had been trying to set up support for (long gone) "int sets", and to import an old version of Zope's Interface package, which doesn't even ship with ZODB. The latter in particular created problems, at least clashing with PythonCAD's Interface package.

POSException

Collector #1488 (TemporaryStorage -- going backward in time). This confusion was really due to that the detail on a ConflictError exception didn't make sense. It called the current revision "was", and the old revision "now". The detail is much more informative now. For example, if the exception said:

ConflictError: database conflict error (oid 0xcb22,
serial was 0x03441422948b4399, now 0x034414228c3728d5)

before, it now says:

ConflictError: database conflict error (oid 0xcb22,
serial this txn started with 0x034414228c3728d5 2002-04-14 20:50:32.863000,
serial currently committed 0x03441422948b4399 2002-04-14 20:50:34.815000)

ConflictError

The undocumented get_old_serial() and get_new_serial() methods were swapped (the first returned the new serial, and the second returned the old serial).

Tools

FileStorage.FileIterator was confused about how to read a transaction's user and description fields, which caused several tools to display binary gibberish for these values.

ZODB.utils.oid_repr() changed to add a leading "0x", and to strip leading zeroes. This is used, e.g., in the detail of a POSKeyError exception, to identify the missing oid. Before, the output was ambiguous. For example, oid 17 was displayed as 0000000000000011. As a Python integer, that's octal 9. Or was it meant to be decimal 11? Or was it meant to be hex? Now it displays as 0x11.

fsrefs.py:

When run with -v, produced tracebacks for objects whose creation was merely undone. This was confusing. Tracebacks are now produced only if there's "a real" problem loading an oid.

If the current revision of object O refers to an object P whose creation has been undone, this is now identified as a distinct case.

Captured and ignored most attempts to stop it via Ctrl+C. Repaired.

Now makes two passes, so that an accurate report can be given of all invalid references.

analyze.py produced spurious "len of unsized object" messages when finding a data record for an object uncreation or version abort. These no longer appear.

fsdump.py's get_pickle_metadata() function (which is used by several tools) was confused about what to do when the ZODB pickle started with a pickle GLOBAL opcode. It actually loaded the class then, which it intends never to do, leading to stray messages on stdout when the class wasn't available, and leading to a strange return value even when it was available (the repr of the type object was returned as "the module name", and an empty string was returned as "the class name"). This has been repaired.

What's new in ZODB3 3.3 beta 2

Release date: 13-Aug-2004

Transaction Managers

Zope3-dev Collector #139: Memory leak involving buckets and connections

The transaction manager internals effectively made every Connection object immortal, except for those explicitly closed. Since typical practice is not to close connections explicitly (and closing a DB happens not to close the connections to it -- although that may change), this caused massive memory leaks when many connections were opened. The transaction manager internals were reworked to use weak references instead, so that connection memory (and other registered synch objects) now get cleaned up when nothing other than the transaction manager knows about them.

Storages

Collector #1327: FileStorage init confused by time travel

If the system clock "went backwards" a long time between the times a FileStorage was closed and reopened, new transaction ids could be smaller than transaction ids already in the storage, violating a key invariant. Now transaction ids are guaranteed to be increasing even when this happens. If time appears to have run backwards at all when a FileStorage is opened, a new message saying so is logged at warning level; if time appears to have run backwards at least 30 minutes, the message is logged at critical level (and you should investigate to find and repair the true cause).

Tools

repozo.py: Thanks to a suggestion from Toby Dickenson, backups (whether incremental or full) are first written to a temp file now, which is fsync'ed at the end, and only after that succeeds is the file renamed to YYYY-MM-DD-HH-MM-SS.ext form. In case of a system crash during a repozo backup, this at least makes it much less likely that a backup file with incomplete or incorrect data will be left behind.

fsrefs.py: Fleshed out the module docstring, and repaired a bug wherein spurious error msgs could be produced after reporting a problem with an unloadable object.

Test suite

Collector #1397: testTimeStamp fails on FreeBSD

The BSD distributions are unique in that their mktime() implementation usually ignores the input tm_isdst value. Test checkFullTimeStamp() was sensitive to this platform quirk.

Reworked the way some of the ZEO tests use threads, so that unittest is more likely to notice the real cause of a failure (which usually occurs in a thread), and less likely to latch on to spurious problems resulting from the real failure.

What's new in ZODB3 3.3 beta 1

Release date: 07-Jun-2004

3.3b1 is the first ZODB release built using the new zpkg tools:

http://zope.org/Members/fdrake/zpkgtools/

This appears to have worked very well. The structure of the tarball release differs from previous releases because of it, and the set of installed files includes some that were not installed in previous releases. That shouldn't create problems, so let us know if it does! We'll fine-tune this for the next release.

BTrees

Fixed bug indexing BTreeItems objects with negative indexes. This caused reverse iteration to return each item twice. Thanks to Casey Duncan for the fix.

ZODB

Methods removed from the database (ZODB.DB.DB) class: cacheStatistics(), cacheMeanAge(), cacheMeanDeac(), and cacheMeanDeal(). These were undocumented, untested, and unused. The first always returned an empty tuple, and the rest always returned None.

When trying to do recovery to a time earlier than that of the most recent full backup, repozo.py failed to find the appropriate files, erroneously claiming "No files in repository before <specified time>". This has been repaired.

Collector #1330: repozo.py -R can create corrupt .fs. When looking for the backup files needed to recreate a Data.fs file, repozo could (unintentionally) include its meta .dat files in the list, or random files of any kind created by the user in the backup directory. These would then get copied verbatim into the reconstructed file, filling parts with junk. Repaired by filtering the file list to include only files with the data extensions repozo.py creates (.fs, .fsz, .deltafs, and .deltafsz). Thanks to James Henderson for the diagnosis.

fsrecover.py couldn't work, because it referenced attributes that no longer existed after the MVCC changes. Repaired that, and added new tests to ensure it continues working.

Collector #1309: The reference counts reported by DB.cacheExtremeDetails() for ghosts were one too small. Thanks to Dieter Maurer for the diagnosis.

Collector #1208: Infinite loop in cPickleCache. If a persistent object had a __del__ method (probably not a good idea regardless, but we don't prevent it) that referenced an attribute of self, the code to deactivate objects in the cache could get into an infinite loop: ghostifying the object could lead to calling its __del__ method, the latter would load the object into cache again to satsify the attribute reference, the cache would again decide that the object should be ghostified, and so on. The infinite loop no longer occurs, but note that objects of this kind still aren't sensible (they're effectively immortal). Thanks to Toby Dickenson for suggesting a nice cure.

What's new in ZODB3 3.3 alpha 3

Release date: 16-Apr-2004

transaction

There is a new transaction package, which provides new interfaces for application code and for the interaction between transactions and resource managers.

The top-level transaction package has functions commit(), abort(), get(), and begin(). They should be used instead of the magic get_transaction() builtin, which will be deprecated. For example:

>>> get_transaction().commit()

should now be written as

>>> import transaction
>>> transaction.commit()

The new API provides explicit transaction manager objects. A transaction manager (TM) is responsible for associating resource managers with a "current" transaction. The default TM, implemented by class ThreadedTransactionManager, assigns each thread its own current transaction. This default TM is available as transaction.manager. The TransactionManager class assigns all threads to the same transaction, and is an explicit replacement for the Connection.setLocalTransaction() method:

A transaction manager instance can be passed as the transaction_manager argument to DB.open(). If you do, the connection will use the specified transaction manager instead of the default TM. The current transaction is obtained by calling get() on a TM. For example:

>>> tm = transaction.TransactionManager()
>>> cn = db.open(transaction_manager=tm)
[...]
>>> tm.get().commit()

The setLocalTransaction() and getTransaction() methods of Connection are deprecated. Use an explicit TM passed via transaction_manager= to DB.open() instead. The setLocalTransaction() method still works, but it returns a TM instead of a Transaction.

A TM creates Transaction objects, which are used for exactly one transaction. Transaction objects still have commit(), abort(), note(), setUser(), and setExtendedInfo() methods.

Resource managers, e.g. Connection or RDB adapter, should use a Transaction's join() method instead of its register() method. An object that calls join() manages its own resources. An object that calls register() expects the TM to manage the objects.

Data managers written against the ZODB 4 transaction API are now supported in ZODB 3.

persistent

A database can now contain persistent weak references. An object that is only reachable from persistent weak references will be removed by pack().

The persistence API now distinguishes between deactivation and invalidation. This change is intended to support objects that can't be ghosts, like persistent classes. Deactivation occurs when a user calls _p_deactivate() or when the cache evicts objects because it is full. Invalidation occurs when a transaction updates the object. An object that can't be a ghost must load new state when it is invalidated, but can ignore deactivation.

Persistent objects can implement a __getnewargs__() method that will be used to provide arguments that should be passed to __new__() when instances (including ghosts) are created. An object that implements __getnewargs__() must be loaded from storage even to create a ghost.

There is new support for writing hooks like __getattr__ and __getattribute__. The new hooks require that user code call special persistence methods like _p_getattr() inside their hook. See the ZODB programming guide for details.

The format of serialized persistent references has changed; that is, the on-disk format for references has changed. The old format is still supported, but earlier versions of ZODB will not be able to read the new format.

ZODB

Closing a ZODB Connection while it is registered with a transaction, e.g. has pending modifications, will raise a ConnnectionStateError. Trying to load objects from or store objects to a closed connection will also raise a ConnnectionStateError.

ZODB connections are synchronized on commit, even when they didn't modify objects. This feature assumes that the thread that opened the connection is also the thread that uses it. If not, this feature will cause problems. It can be disabled by passing synch=False to open().

New broken object support.

New add() method on Connection. User code should not assign the _p_jar attribute of a new persistent object directly; a deprecation warning is issued in this case.

Added a get() method to Connection as a preferred synonym for __getitem__().

Several methods and/or specific optional arguments of methods have been deprecated. The cache_deactivate_after argument used by DB() and Connection() is deprecated. The DB methods getCacheDeactivateAfter(), getVersionCacheDeactivateAfter(), setCacheDeactivateAfter(), and setVersionCacheDeactivateAfter() are also deprecated.

The old-style undo() method was removed from the storage API, and transactionalUndo() was renamed to undo().

The BDBStorages are no longer distributed with ZODB.

Fixed a serious bug in the new pack implementation. If pack was called on the storage and passed a time earlier than a previous pack time, data could be lost. In other words, if there are any two pack calls, where the time argument passed to the second call was earlier than the first call, data loss could occur. The bug was fixed by causing the second call to raise a StorageError before performing any work.

Fixed a rare bug in pack: if a pack started during a small window of time near the end of a concurrent transaction's commit, it was possible for the pack attempt to raise a spurious

CorruptedError: ... transaction with checkpoint flag set

exception. This did no damage to the database, or to the transaction in progress, but no pack was performed then.

By popular demand, FileStorage.pack() no longer propagates a

FileStorageError: The database has already been packed to a later time or no changes have been made since the last pack

exception. Instead that message is logged (at INFO level), and the pack attempt simply returns then (no pack is performed).

ZEO

Fixed a bug that prevented the -m / --monitor argument from working.

zdaemon

Added a -m / --mask option that controls the umask of the subprocess.

zLOG

The zLOG backend has been removed. zLOG is now just a facade over the standard Python logging package. Environment variables like STUPID_LOG_FILE are no longer honored. To configure logging, you need to follow the directions in the logging package documentation. The process is currently more complicated than configured zLOG. See test.py for an example.

ZConfig

This release of ZODB contains ZConfig 2.1.

More documentation has been written.

Make sure keys specified as attributes of the <default> element are converted by the appropriate key type, and are re-checked for derived sections.

Refactored the ZConfig.components.logger schema components so that a schema can import just one of the "eventlog" or "logger" sections if desired. This can be helpful to avoid naming conflicts.

Added a reopen() method to the logger factories.

Always use an absolute pathname when opening a FileHandler.

Miscellaneous

The layout of the ZODB source release has changed. All the source code is contained in a src subdirectory. The primary motivation for this change was to avoid confusion caused by installing ZODB and then testing it interactively from the source directory; the interpreter would find the uncompiled ZODB package in the source directory and report an import error.

A reference-counting bug was fixed, in the logic calling a modified persistent object's data manager's register() method. The primary symptom was rare assertion failures in Python's cyclic garbage collection.

The Connection class's onCommitAction() method was removed.

Some of the doc strings in ZODB are now written for processing by epydoc.

Several new test suites were written using doctest instead of the standard unittest TestCase framework.

MappingStorage now implements getTid().

ThreadedAsync: Provide a way to shutdown the servers using an exit status.

The mkzeoinstance script looks for a ZODB installation, not a Zope installation. The received wisdom is that running a ZEO server without access to the appserver code avoids many mysterious problems.

What's new in ZODB3 3.3 alpha 2

Release date: 06-Jan-2004

This release contains a major overhaul of the persistence machinery, including some user-visible changes. The Persistent base class is now a new-style class instead of an ExtensionClass. The change enables the use of features like properties with persistent object classes. The Persistent base class is now contained in the persistent package.

The Persistence package is included for backwards compatibility. The Persistence package is used by Zope to provide special ExtensionClass-compatibility features like a non-C3 MRO and an __of__ method. ExtensionClass is not included with this release of ZODB3. If you use the Persistence package, it will print a warning and import Persistent from persistent.

In short, the new persistent package is recommended for non-Zope applications. The following dotted class names are now preferred over earlier names:

  • persistent.Persistent
  • persistent.list.PersistentList
  • persistent.mapping.PersistentMapping
  • persistent.TimeStamp

The in-memory, per-connection object cache (pickle cache) was changed to participate in garbage collection. This should reduce the number of memory leaks, although we are still tracking a few problems.

Multi-version concurrency control

ZODB now supports multi-version concurrency control (MVCC) for storages that support multiple revisions. FileStorage and BDBFullStorage both support MVCC. In short, MVCC means that read conflicts should almost never occur. When an object is modified in one transaction, other concurrent transactions read old revisions of the object to preserve consistency. In earlier versions of ZODB, any access of the modified object would raise a ReadConflictError.

The ZODB internals changed significantly to accommodate MVCC. There are relatively few user visible changes, aside from the lack of read conflicts. It is possible to disable the MVCC feature using the mvcc keyword argument to the DB open() method, ex.: db.open(mvcc=False).

ZEO

Changed the ZEO server and control process to work with a single configuration file; this is now the default way to configure these processes. (It's still possible to use separate configuration files.) The ZEO configuration file can now include a "runner" section used by the control process and ignored by the ZEO server process itself. If present, the control process can use the same configuration file.

Fixed a performance problem in the logging code for the ZEO protocol. The logging code could call repr() on arbitrarily long lists, even though it only logged the first 60 bytes; worse, it called repr() even if logging was currently disabled. Fixed to call repr() on individual elements until the limit is reached.

Fixed a bug in zrpc (when using authentication) where the MAC header wasn't being read for large messages, generating errors while unpickling commands sent over the wire. Also fixed the zeopasswd.py script, added testcases and provided a more complete commandline interface.

Fixed a misuse of the _map variable in zrpc Connectio objects, which are also asyncore.dispatcher objects. This allows ZEO to work with CVS Python (2.4). _map is used to indicate whether the dispatcher users the default socket_map or a custom socket_map. A recent change to asyncore caused it to use _map in its add_channel() and del_channel() methods, which presumes to be a bug fix (may get ported to 2.3). That causes our dubious use of _map to be a problem, because we also put the Connections in the global socket_map. The new asyncore won't remove it from the global socket map, because it has a custom _map.

The prefix used for log messages from runzeo.py was changed from RUNSVR to RUNZEO.

Miscellaneous

ReadConflictError objects now have an ignore() method. Normally, a transaction that causes a read conflict can't be committed. If the exception is caught and its ignore() method called, the transaction can be committed. Application code may need this in advanced applications.

What's new in ZODB3 3.3 alpha 1

Release date: 17-Jul-2003

New features of Persistence

The Persistent base class is a regular Python type implemented in C. It should be possible to create new-style classes that inherit from Persistent, and, thus, use all the new Python features introduced in Python 2.2 and 2.3.

The __changed__() method on Persistent objects is no longer supported.

New features in BTrees

BTree, Bucket, TreeSet and Set objects are now iterable objects, playing nicely with the iteration protocol introduced in Python 2.2, and can be used in any context that accepts an iterable object. As for Python dicts, the iterator constructed for BTrees and Buckets iterates over the keys.

>>> from BTrees.OOBTree import OOBTree
>>> b = OOBTree({"one": 1, "two": 2, "three": 3, "four": 4})
>>> for key in b: # iterates over the keys
...    print key
four
one
three
two
>>> list(enumerate(b))
[(0, 'four'), (1, 'one'), (2, 'three'), (3, 'two')]
>>> i = iter(b)
>>> i.next()
'four'
>>> i.next()
'one'
>>> i.next()
'three'
>>> i.next()
'two'
>>>

As for Python dicts in 2.2, BTree and Bucket objects have new .iterkeys(), .iteritems(), and .itervalues() methods. TreeSet and Set objects have a new .iterkeys() method. Unlike as for Python dicts, these new methods accept optional min and max arguments to effect range searches. While Bucket.keys() produces a list, Bucket.iterkeys() produces an iterator, and similarly for Bucket values() versus itervalues(), Bucket items() versus iteritems(), and Set keys() versus iterkeys(). The iter{keys,values,items} methods of BTrees and the iterkeys() method of Treesets also produce iterators, while their keys() (etc) methods continue to produce BTreeItems objects (a form of "lazy" iterator that predates Python 2.2's iteration protocol).

>>> sum(b.itervalues())
10
>>> zip(b.itervalues(), b.iterkeys())
[(4, 'four'), (1, 'one'), (3, 'three'), (2, 'two')]
>>>

BTree, Bucket, TreeSet and Set objects also implement the __contains__ method new in Python 2.2, which means that testing for key membership can be done directly now via the "in" and "not in" operators:

>>> "won" in b
False
>>> "won" not in b
True
>>> "one" in b
True
>>>

All old and new range-search methods now accept keyword arguments, and new optional excludemin and excludemax keyword arguments. The new keyword arguments allow doing a range search that's exclusive at one or both ends (doesn't include min, and/or doesn't include max).

>>> list(b.keys())
['four', 'one', 'three', 'two']
>>> list(b.keys(max='three'))
['four', 'one', 'three']
>>> list(b.keys(max='three', excludemax=True))
['four', 'one']
>>>

Other improvements

The exceptions generated by write conflicts now contain the name of the conflicted object's class. This feature requires support for the storage. All the standard storages support it.

What's new in ZODB3 3.2

Release date: 08-Oct-2003

Nothing has changed since release candidate 1.

What's new in ZODB3 3.2 release candidate 1

Release date: 01-Oct-2003

Added a summary to the Doc directory. There are several new documents in the 3.2 release, including "Using zdctl and zdrun to manage server processes" and "Running a ZEO Server HOWTO."

Fixed ZEO's protocol negotiation mechanism so that a client ZODB 3.1 can talk to a ZODB 3.2 server.

Fixed a memory leak in the ZEO server. The server was leaking a few KB of memory per connection.

Fixed a memory leak in the ZODB object cache (cPickleCache). The cache did not release two references to its Connection, causing a large cycle of objects to leak when a database was closed.

Fixed a bug in the ZEO code that caused it to leak socket objects on Windows. Specifically, fix the trigger mechanism so that both sockets created for a trigger are closed.

Fixed a bug in the ZEO storage server that caused it to leave temp files behind. The CommitLog class contains a temp file, but it was not closing the file.

Changed the order of setuid() and setgid() calls in zdrun, so that setgid() is called first.

Added a timeout to the ZEO test suite that prevents hangs. The test suite creates ZEO servers with randomly assigned ports. If the port happens to be in use, the test suite would hang because the ZEO client would never stop trying to connect. The fix will cause the test to fail after a minute, but should prevent the test runner from hanging.

The logging package was updated to include the latest version of the logging package from Python CVS. Note that this package is only installed for Python 2.2. In later versions of Python, it is available in the Python standard library.

The ZEO1 directory was removed from the source distribution. ZEO1 is not supported, and we never intended to include it in the release.

What's new in ZODB3 3.2 beta 3

Release date: 23-Sep-2003

Note: The changes listed for this release include changes also made in ZODB 3.1.x releases and ported to the 3.2 release.

This version of ZODB 3.2 is not compatible with Python 2.1. Early versions were explicitly designed to be compatible with Zope 2.6. That plan has been dropped, because Zope 2.7 is already in beta release.

Several of the classes in ZEO and ZODB now inherit from object, making them new-style classes. The primary motivation for the change was to make it easier to debug memory leaks. We don't expect any behavior to change as a result.

A new feature to allow removal of connection pools for versions was ported from Zope 2.6. This feature is needed by Zope to avoid denial of service attacks that allow a client to create an arbitrary number of version pools.

Fixed several critical ZEO bugs.

  • If several client transactions were blocked waiting for the storage and one of the blocked clients disconnected, the server would attempt to restart one of the other waiting clients. Since the disconnected client did not have the storage lock, this could lead to deadlock. It could also cause the assertion "self._client is None" to fail.
  • If a storage server fails or times out between the vote and the finish, the ZEO cache could get populated with objects that didn't make it to the storage server.
  • If a client loses its connection to the server near the end of a transaction, it is now guaranteed to get a ClientDisconnected error even if it reconnects before the transaction finishes. This is necessary because the server will always abort the transaction. In some cases, the client would never see an error for the aborted transaction.
  • In tpc_finish(), reordered the calls so that the server's tpc_finish() is called (and must succeed) before we update the ZEO client cache.
  • The storage name is now prepended to the sort key, to ensure a unique global sort order if storages are named uniquely. This can prevent deadlock in some unusual cases.

Fixed several serious flaws in the implementation of the ZEO authentication protocol.

  • The smac layer would accept a message without a MAC even after the session key was established.
  • The client never initialized its session key, so it never checked incoming messages or created MACs for outgoing messags.
  • The smac layer used a single HMAC instance for sending and receiving messages. This approach could only work if client and server were guaranteed to process all messages in the same total order, which could only happen in simple scenarios like unit tests.

Fixed a bug in ExtensionClass when comparing ExtensionClass instances. The code could raise RuntimeWarning under Python 2.3, and produce incorrect results on 64-bit platforms.

Fixed bug in BDBStorage that could lead to DBRunRecoveryErrors when a transaction was aborted after performing operations like commit version or undo that create new references to existing pickles.

Fixed a bug in Connection.py that caused it to fail with an AttributeError if close() was called after the database was closed.

The test suite leaves fewer log files behind, although it still leaves a lot of junk. The test.py script puts each tests temp files in a separate directory, so it is easier to see which tests are causing problems. Unfortunately, it is still to tedious to figure out why the identified tests are leaving files behind.

This release contains the latest and greatest version of the BDBStorage. This storage has still not seen testing in a production environment, but it represents the current best design and most recent code culled from various branches where development has occurred.

The Tools directory contains a number of small improvements, a few new tools, and README.txt that catalogs the tools. Many of the tools are installed by setup.py; those scripts will now have a #! line set automatically on Unix.

Fixed bugs in Tools/repozo.py, including a timing-dependent one that could cause the following invocation of repozo to do a full backup when an incremental backup would have sufficed.

A pair of new scripts from Jim Fulton can be used to synthesize workloads and measure ZEO performance: see zodbload.py and zeoserverlog.py in the Tools directory. Note that these require Zope.

Tools/checkbtrees.py was strengthened in two ways:

  • In addition to running the _check() method on each BTree B found, BTrees.check.check(B) is also run. The check() function was written after checkbtrees.py, and identifies kinds of damage B._check() cannot find.
  • Cycles in the object graph no longer lead to unbounded output. Note that preventing this requires remembering the oid of each persistent object found, which increases the memory needed by the script.

What's new in ZODB3 3.2 beta 2

Release date: 16-Jun-2003

Fixed critical race conditions in ZEO's cache consistency code that could cause invalidations to be lost or stale data to be written to the cache. These bugs can lead to data loss or data corruption. These bugs are relatively unlikely to be provoked in sites with few conflicts, but the possibility of failure existed any time an object was loaded and stored concurrently.

Fixed a bug in conflict resolution that failed to ghostify an object if it was involved in a conflict. (This code may be redundant, but it has been fixed regardless.)

The ZEO server was fixed so that it does not perform any I/O until all of a transactions' invalidations are queued. If it performs I/O in the middle of sending invalidations, it would be possible to overlap a load from a client with the invalidation being sent to it.

The ZEO cache now handles invalidations atomically. This is the same sort of bug that is described in the 3.1.2b1 section below, but it affects the ZEO cache.

Fixed several serious bugs in fsrecover that caused it to fail catastrophically in certain cases because it thought it had found a checkpoint (status "c") record when it was in the middle of the file.

Two new features snuck into this beta release.

The ZODB.transact module provides a helper function that converts a regular function or method into a transactional one.

The ZEO client cache now supports Adaptable Persistence (APE). The cache used to expect that all OIDs were eight bytes long.

What's new in ZODB3 3.2 beta 1

Release date: 30-May-2003

ZODB

Invalidations are now processed atomically. Each transaction will see all the changes caused by an earlier transaction or none of them. Before this patch, it was possible for a transaction to see invalid data because it saw only a subset of the invalidations. This is the most likely cause of reported BTrees corruption, where keys were stored in the wrong bucket. When a BTree bucket splits, the bucket and the bucket's parent are both modified. If a transaction sees the invalidation for the bucket but not the parent, the BTree in memory will be internally inconsistent and keys can be put in the wrong bucket. The atomic invalidation fix prevents this problem.

A number of minor reference count fixes in the object cache were fixed. That's the cPickleCache.c file.

It was possible for a transaction that failed in tpc_finish() to lose the traceback that caused the failure. The transaction code was fixed to report the original error as well as any errors that occur while trying to recover from the original error.

The "other" argument to copyTransactionsFrom() only needs to have an .iterator() method. For convenience, change FileStorage's and BDBFullStorage's iterator to have this method, which just returns self.

Mount points are now visible from mounted objects.

Fixed memory leak involving database connections and caches. When a connection or database was closed, the cache and database leaked, because of a circular reference involving the cache. Fixed the cache to explicitly clear out its contents when its connection is closed.

The ZODB cache has fewer methods. It used to expose methods that could mutate the dictionary, which allowed users to violate internal invariants.

ZConfig

It is now possible to configure ZODB databases and storages and ZEO servers using ZConfig.

ZEO & zdaemon

ZEO now supports authenticated client connections. The default authentication protocol uses a hash-based challenge-response protocol to prove identity and establish a session key for message authentication. The architecture is pluggable to allow third-parties to developer better authentication protocols.

There is a new HOWTO for running a ZEO server. The draft in this release is incomplete, but provides more guidance than previous releases. See the file Doc/ZEO/howto.txt.

The ZEO storage server's transaction timeout feature was refactored and made slightly more rebust.

A new ZEO utility script, ZEO/mkzeoinst.py, was added. This creates a standard directory structure and writes a configuration file with mostly default values, and a bootstrap script that can be used to manage and monitor the server using zdctl.py (see below).

Much work was done to improve zdaemon's zdctl.py and zdrun.py scripts. (In the alpha 1 release, zdrun.py was called zdaemon.py, but installing it in <prefix>/bin caused much breakage due to the name conflict with the zdaemon package.) Together with the new mkzeoinst.py script, this makes controlling a ZEO server a breeze.

A ZEO client will not read from its cache during cache verification. This fix was necessary to prevent the client from reading inconsistent data.

The isReadOnly() method of a ZEO client was fixed to return the false when the client is connected to a read-only fallback server.

The sync() method of ClientStorage and the pending() method of a zrpc connection now do both input and output.

The short_repr() function used to generate log messages was fixed so that it does not blow up creating a repr of very long tuples.

Storages

FileStorage has a new pack() implementation that fixes several reported problems that could lead to data loss.

Two small bugs were fixed in DemoStorage. undoLog() did not handle its arguments correctly and pack() could accidentally delete objects created in versions.

Fixed trivial bug in fsrecover that prevented it from working at all.

FileStorage will use fsync() on Windows starting with Python 2.2.3.

FileStorage's commit version was fixed. It used to stop after the first object, leaving all the other objects in the version.

BTrees

Trying to store an object of a non-integer type into an IIBTree or OIBTree could leave the bucket in a variety of insane states. For example, trying

b[obj] = "I'm a string, not an integer"

where b is an OIBTree. This manifested as a refcount leak in the test suite, but could have been much worse (most likely in real life is that a seemingly arbitrary existing key would "go missing").

When deleting the first child of a BTree node with more than one child, a reference to the second child leaked. This could cause the entire bucket chain to leak (not be collected as garbage despite not being referenced anymore).

Other minor BTree leak scenarios were also fixed.

Tools

New tool zeoqueue.py for parsing ZEO log files, looking for blocked transactions.

New tool repozo.py (originally by Anthony Baxter) for performing incremental backups of Data.fs files.

The fsrecover.py script now does a better job of recovering from errors the occur in the middle of a transaction record. Fixed several bugs that caused partial or total failures in earlier versions.

What's new in ZODB3 3.2 alpha 1

Release date: 17-Jan-2003

Most of the changes in this release are performance and stability improvements to ZEO. A major packaging change is that there won't be a separate ZEO release. The new ZConfig is a noteworthy addtion (see below).

ZODB

An experimental new transaction API was added. The Connection class has a new method, setLocalTransaction(). ZODB applications can call this method to bind transactions to connections rather than threads. This is especially useful for GUI applications, which often have only one thread but multiple independent activities within that thread (generally one per window). Thanks to Christian Reis for championing this feature.

Applications that take advantage of this feature should not use the get_transaction() function. Until now, ZODB itself sometimes assumed get_transaction() was the only way to get the transaction. Minor corrections have been added. The ZODB test suite, on the other hand, can continue to use get_transaction(), since it is free to assume that transactions are bound to threads.

ZEO

There is a new recommended script for starting a storage server. We recommend using ZEO/runzeo.py instead of ZEO/start.py. The start.py script is still available in this release, but it will no longer be maintained and will eventually be removed.

There is a new zdaemon implementation. This version is a separate script that runs an arbitrary daemon. To run the ZEO server as a daemon, you would run "zdrun.py runzeo.py". There is also a simple shell, zdctl.py, that can be used to manage a daemon. Try "zdctl.py -p runzeo.py".

There is a new version of the ZEO protocol in this release and a first stab at protocol negotiation. (It's a first stab because the protocol checking supporting in ZODB 3.1 was too primitive to support anything better.) A ZODB 3.2 ZEO client can talk to an old server, but a ZODB 3.2 server can't talk to an old client. It's safe to upgrade all the clients first and upgrade the server last. The ZEO client cache format changed, so you'll need to delete persistent caches before restarting clients.

The ZEO cache verification protocol was revised to require many fewer messages in cases where a client or server restarts quickly.

The performance of full cache verification has improved dramatically. Measurements from Jim were somewhere in 2x-5x. The implementation was fixed to use the very-fast getSerial() method on the storage instead of the comparatively slow load().

The ZEO server has an optional timeout feature that will abort a connection that does not commit within a certain amount of time. The timeout works by closing the socket the client is using, causing both client and server to abort the transaction and continue. This is a drastic step, but can be useful to prevent a hung client or other bug from blocking a server indefinitely.

A bug was fixed in the ZEO protocol that allowed clients to read stale cache data while cache verification was being performed. The fixed version prevents the client from using the storage until after verification completes.

The ZEO server has an experimental monitoring interface that reports usage statistics for the storage server including number of connected clients and number of transactions active and committed. It can be enabled by passing the -m flag to runsvr.py.

The ZEO ClientStorage no longer supports the environment variables CLIENT_HOME, INSTANCE_HOME, or ZEO_CLIENT.

The ZEO1 package is still included with this release, but there is no longer an option to install it.

BTrees

The BTrees package now has a check module that inspects a BTree to check internal invariants. Bugs in older versions of the code code leave a BTree in an inconsistent state. Calling BTrees.check.check() on a BTree object should verify its consistency. (See the NEWS section for 3.1 beta 1 below to for the old BTrees bugs.)

Fixed a rare conflict resolution problem in the BTrees that could cause an segfault when the conflict resolution resulted in any empty bucket.

Installation

The distutils setup now installs several Python scripts. The runzeo.py and zdrun.py scripts mentioned above and several fsXXX.py scripts from the Tools directory.

The test.py script does not run all the ZEO tests by default, because the ZEO tests take a long time to run. Use --all to run all the tests. Otherwise a subset of the tests, mostly using MappingStorage, are run.

Storages

There are two new storages based on Sleepycat's BerkeleyDB in the BDBStorage package. Barry will have to write more here, because I don't know how different they are from the old bsddb3Storage storages. See Doc/BDBStorage.txt for more information.

It now takes less time to open an existing FileStorage. The FileStorage uses a BTree-based index that is faster to pickle and unpickle. It also saves the index periodically so that subsequent opens will go fast even if the storage was not closed cleanly.

Misc

The new ZConfig package, which will be used by Zope and ZODB, is included. ZConfig provides a configuration syntax, similar to Apache's syntax. The package can be used to configure the ZEO server and ZODB databases. See the module ZODB.config for functions to open the database from configuration. See ZConfig/doc for more info.

The zLOG package now uses the logging package by Vinay Sajip, which will be included in Python 2.3.

The Sync extension was removed from ExtensionClass, because it was not used by ZODB.

What's new in ZODB3 3.1.4?

Release date: 11-Sep-2003

A new feature to allow removal of connection pools for versions was ported from Zope 2.6. This feature is needed by Zope to avoid denial of service attacks that allow a client to create an arbitrary number of version pools.

A pair of new scripts from Jim Fulton can be used to synthesize workloads and measure ZEO performance: see zodbload.py and zeoserverlog.py in the Tools directory. Note that these require Zope.

Tools/checkbtrees.py was strengthened in two ways:

  • In addition to running the _check() method on each BTree B found, BTrees.check.check(B) is also run. The check() function was written after checkbtrees.py, and identifies kinds of damage B._check() cannot find.
  • Cycles in the object graph no longer lead to unbounded output. Note that preventing this requires remembering the oid of each persistent object found, which increases the memory needed by the script.

What's new in ZODB3 3.1.3?

Release date: 18-Aug-2003

Fixed several critical ZEO bugs.

  • If a storage server fails or times out between the vote and the finish, the ZEO cache could get populated with objects that didn't make it to the storage server.
  • If a client loses its connection to the server near the end of a transaction, it is now guaranteed to get a ClientDisconnected error even if it reconnects before the transaction finishes. This is necessary because the server will always abort the transaction. In some cases, the client would never see an error for the aborted transaction.
  • In tpc_finish(), reordered the calls so that the server's tpc_finish() is called (and must succeed) before we update the ZEO client cache.
  • The storage name is now prepended to the sort key, to ensure a unique global sort order if storages are named uniquely. This can prevent deadlock in some unusual cases.

A variety of fixes and improvements to Berkeley storage (aka BDBStorage) were back-ported from ZODB 4. This release now contains the most current version of the Berkeley storage code. Many tests have been back-ported, but not all.

Modified the Windows tests to wait longer at the end of ZEO tests for the server to shut down. Before Python 2.3, there is no waitpid() on Windows, and, thus, no way to know if the server has shut down. The change makes the Windows ZEO tests much less likely to fail or hang, at the cost of increasing the time needed to run the tests.

Fixed a bug in ExtensionClass when comparing ExtensionClass instances. The code could raise RuntimeWarning under Python 2.3, and produce incorrect results on 64-bit platforms.

Fixed bugs in Tools/repozo.py, including a timing-dependent one that could cause the following invocation of repozo to do a full backup when an incremental backup would have sufficed.

Added Tools/README.txt that explains what each of the scripts in the Tools directory does.

There were many small changes and improvements to the test suite.

What's new in ZODB3 3.1.2 final?

Fixed bug in FileStorage pack that caused it to fail if it encountered an old undo record (status "u").

Fixed several bugs in FileStorage pack that could cause OverflowErrors for storages > 2 GB.

Fixed memory leak in TimeStamp.laterThan() that only occurred when it had to create a new TimeStamp.

Fixed two BTree bugs that were fixed on the head a while ago:

  • bug in fsBTree that would cause byValue searches to end early. (fsBTrees are never used this way, but it was still a bug.)
  • bug that lead to segfault if BTree was mutated via deletion while it was being iterated over.

What's new in ZODB3 3.1.2 beta 2?

Fixed critical race conditions in ZEO's cache consistency code that could cause invalidations to be lost or stale data to be written to the cache. These bugs can lead to data loss or data corruption. These bugs are relatively unlikely to be provoked in sites with few conflicts, but the possibility of failure existed any time an object was loaded and stored concurrently.

Fixed a bug in conflict resolution that failed to ghostify an object if it was involved in a conflict. (This code may be redundant, but it has been fixed regardless.)

The ZEO server was fixed so that it does not perform any I/O until all of a transactions' invalidations are queued. If it performs I/O in the middle of sending invalidations, it would be possible to overlap a load from a client with the invalidation being sent to it.

The ZEO cache now handles invalidations atomically. This is the same sort of bug that is described in the 3.1.2b1 section below, but it affects the ZEO cache.

Fixed several serious bugs in fsrecover that caused it to fail catastrophically in certain cases because it thought it had found a checkpoint (status "c") record when it was in the middle of the file.

What's new in ZODB3 3.1.2 beta 1?

ZODB

Invalidations are now processed atomically. Each transaction will see all the changes caused by an earlier transaction or none of them. Before this patch, it was possible for a transaction to see invalid data because it saw only a subset of the invalidations. This is the most likely cause of reported BTrees corruption, where keys were stored in the wrong bucket. When a BTree bucket splits, the bucket and the bucket's parent are both modified. If a transaction sees the invalidation for the bucket but not the parent, the BTree in memory will be internally inconsistent and keys can be put in the wrong bucket. The atomic invalidation fix prevents this problem.

A number of minor reference count fixes in the object cache were fixed. That's the cPickleCache.c file.

It was possible for a transaction that failed in tpc_finish() to lose the traceback that caused the failure. The transaction code was fixed to report the original error as well as any errors that occur while trying to recover from the original error.

ZEO

A ZEO client will not read from its cache during cache verification. This fix was necessary to prevent the client from reading inconsistent data.

The isReadOnly() method of a ZEO client was fixed to return the false when the client is connected to a read-only fallback server.

The sync() method of ClientStorage and the pending() method of a zrpc connection now do both input and output.

The short_repr() function used to generate log messages was fixed so that it does not blow up creating a repr of very long tuples.

Storages

FileStorage has a new pack() implementation that fixes several reported problems that could lead to data loss.

Two small bugs were fixed in DemoStorage. undoLog() did not handle its arguments correctly and pack() could accidentally delete objects created in versions.

Fixed trivial bug in fsrecover that prevented it from working at all.

FileStorage will use fsync() on Windows starting with Python 2.2.3.

FileStorage's commit version was fixed. It used to stop after the first object, leaving all the other objects in the version.

BTrees

Trying to store an object of a non-integer type into an IIBTree or OIBTree could leave the bucket in a variety of insane states. For example, trying

b[obj] = "I'm a string, not an integer"

where b is an OIBTree. This manifested as a refcount leak in the test suite, but could have been much worse (most likely in real life is that a seemingly arbitrary existing key would "go missing").

When deleting the first child of a BTree node with more than one child, a reference to the second child leaked. This could cause the entire bucket chain to leak (not be collected as garbage despite not being referenced anymore).

Other minor BTree leak scenarios were also fixed.

Other

Comparing a Missing.Value object to a C type that provide its own comparison operation could lead to a segfault when the Missing.Value was on the right-hand side of the comparison operator. The Missing class was fixed so that its coercion and comparison operations are safe.

Tools

Four tools are now installed by setup.py: fsdump.py, fstest.py, repozo.py, and zeopack.py.

What's new in ZODB3 3.1.1 final?

Release date: 11-Feb-2003

Tools

Updated repozo.py tool

What's new in ZODB3 3.1.1 beta 2?

Release date: 03-Feb-2003

The Transaction "hosed" feature is disabled in this release. If a transaction fails during the tpc_finish() it is not possible, in general, to know whether the storage is in a consistent state. For example, a ZEO server may commit the data and then fail before sending confirmation of the commit to the client. If multiple storages are involved in a transaction, the problem is exacerbated: One storage may commit the data while another fails to commit. In previous versions of ZODB, the database would set a global "hosed" flag that prevented any other transaction from committing until an administrator could check the status of the various failed storages and ensure that the database is in a consistent state. This approach favors data consistency over availability. The new approach is to log a panic but continue. In practice, availability seems to be more important than consistency. The failure mode is exceedingly rare in either case.

The BTrees-based fsIndex for FileStorage is enabled. This version of the index is faster to load and store via pickle and uses less memory to store keys. We had intended to enable this feature in an earlier release, but failed to actually do it; thus, it's getting enabled as a bug fix now.

Two rare bugs were fixed in BTrees conflict resolution. The most probable symptom of the bug would have been a segfault. The bugs were found via synthetic stress tests rather than bug reports.

A value-based consistency checker for BTrees was added. See the module BTrees.check for the checker and other utilities for working with BTrees.

A new script called repozo.py was added. This script, originally written by Anthony Baxter, provides an incremental backup scheme for FileStorage based storages.

zeopack.py has been fixed to use a read-only connection.

Various small autopack-related race conditions have been fixed in the Berkeley storage implementations. There have been some table changes to the Berkeley storages so any storage you created in 3.1.1b1 may not work. Part of these changes was to add a storage version number to the schema so these types of incompatible changes can be avoided in the future.

Removed the chance of bogus warnings in the FileStorage iterator.

ZEO

The ZEO version number was bumped to 2.0.2 on account of the following minor feature additions.

The performance of full cache verification has improved dramatically. Measurements from Jim were somewhere in 2x-5x. The implementation was fixed to use the very-fast getSerial() method on the storage instead of the comparatively slow load().

The ZEO server has an optional timeout feature that will abort a connection that does not commit within a certain amount of time. The timeout works by closing the socket the client is using, causing both client and server to abort the transaction and continue. This is a drastic step, but can be useful to prevent a hung client or other bug from blocking a server indefinitely.

If a client was disconnected during a transaction, the tpc_abort() call did not properly reset the internal state about the transaction. The bug caused the next transaction to fail in its tpc_finish(). Also, any ClientDisconnected exceptions raised during tpc_abort() are ignored.

ZEO logging has been improved by adding more logging for important events, and changing the logging level for existing messages to a more appropriate level (usually lower).

What's new in ZODB3 3.1.1 beta 1?

Release date: 10-Dev-2002

It was possible for earlier versions of ZODB to deadlock when using multiple storages. If multiple transactions committed concurrently and both transactions involved two or more shared storages, deadlock was possible. This problem has been fixed by introducing a sortKey() method to the transaction and storage APIs that is used to define an ordering on transaction participants. This solution will prevent deadlocks provided that all transaction participants that use locks define a valid sortKey() method. A warning is raised if a participant does not define sortKey(). For backwards compatibility, BaseStorage provides a sortKey() that uses __name__.

Added code to ThreadedAsync/LoopCallback.py to work around a bug in asyncore.py: a handled signal can cause unwanted reads to happen.

A bug in FileStorage related to object uncreation was fixed. If an a transaction that created an object was undone, FileStorage could write a bogus data record header that could lead to strange errors if the object was loaded. An attempt to load an uncreated object now raises KeyError, as expected.

The restore() implementation in FileStorage wrote incorrect backpointers for a few corner cases involving versions and undo. It also failed if the backpointer pointed to a record that was before the pack time. These specific bugs have been fixed and new test cases were added to cover them.

A bug was fixed in conflict resolution that raised a NameError when a class involved in a conflict could not be loaded. The bug did not affect correctness, but prevent ZODB from caching the fact that the class was unloadable. A related bug prevented spurious AttributeErrors when a class could not be loaded. It was also fixed.

The script Tools/zeopack.py was fixed to work with ZEO 2. It was untested and had two silly bugs.

Some C extensions included standard header files before including Python.h, which is not allowed. They now include Python.h first, which eliminates compiler warnings in certain configurations.

The BerkeleyDB based storages have been merged from the trunk, providing a much more robust version of the storages. They are not backwards compatible with the old storages, but the decision was made to update them in this micro release because the old storages did not work for all practical purposes. For details, see Doc/BDBStorage.txt.

What's new in ZODB3 3.1 final?

Release date: 28-Oct-2002

If an error occurs during conflict resolution, the store will silently catch the error, log it, and continue as if the conflict was unresolvable. ZODB used to behave this way, and the change to catch only ConflictError was causing problems in deployed systems. There are a lot of legitimate errors that should be caught, but it's too close to the final release to make the substantial changes needed to correct this.

What's new in ZODB3 3.1 beta 3?

Release date: 21-Oct-2002

A small extension was made to the iterator protocol. The Record objects, which are returned by the per-transaction iterators, contain a new data_txn attribute. It is None, unless the data contained in the record is a logical copy of an earlier transaction's data. For example, when transactional undo modifies an object, it creates a logical copy of the earlier transaction's data. Note that this provide a stronger statement about consistency than whether the data in two records is the same; it's possible for two different updates to an object to coincidentally have the same data.

The restore() method was extended to take the data_txn attribute mentioned above as an argument. FileStorage uses the new argument to write a backpointer if possible.

A few bugs were fixed.

The setattr slot of the cPersistence C API was being initialized to NULL. The proper initialization was restored, preventing crashes in some applications with C extensions that used persistence.

The return value of TimeStamp's __cmp__ method was clipped to return only 1, 0, -1.

The restore() method was fixed to write a valid backpointer if the update being restored is in a version.

Several bugs and improvements were made to zdaemon, which can be used to run the ZEO server. The parent now forwards signals to the child as intended. Pidfile handling was improved and the trailing newline was omitted.

What's new in ZODB3 3.1 beta 2?

Release date: 4-Oct-2002

A few bugs have been fixed, some that were found with the help of Neal Norwitz's PyChecker.

The zeoup.py tool has been fixed to allow connecting to a read-only storage, when the --nowrite option is given.

Casey Duncan fixed a few bugs in the recent changes to undoLog().

The fstest.py script no longer checks that each object modified in a transaction has a serial number that matches the transaction id. This invariant is no longer maintained; several new features in the 3.1 release depend on it.

The ZopeUndo package was added. If ZODB3 is being used to run a ZEO server that will be used with Zope, it is usually best if the server and the Zope client don't share any software. The Zope undo framework, however, requires that a Prefix object be passed between client and server. To support this use, ZopeUndo was created to hold the Prefix object.

Many bugs were fixed in ZEO, and a couple of features added. See ZEO-NEWS.txt for details.

The ZODB guide included in the Doc directory has been updated. It is still incomplete, but most of the references to old ZODB packages have been removed. There is a new section that briefly explains how to use BTrees.

The zeoup.py tool connects using a read-only connection when --nowrite is specifified. This feature is useful for checking on read-only ZEO servers.

What's new in ZODB3 3.1 beta 1?

Release date: 12-Sep-2002

We've changed the name and version number of the project, but it's still the same old ZODB. There have been a lot of changes since the last release.

New ZODB cache

Toby Dickenson implemented a new Connection cache for ZODB. The cache is responsible for pointer swizzling (translating between oids and Python objects) and for keeping recently used objects in memory. The new cache is a big improvement over the old cache. It strictly honors its size limit, where size is specified in number of objects, and it evicts objects in least recently used (LRU) order.

Users should take care when setting the cache size, which has a default value of 400 objects. The old version of the cache often held many more objects than its specified size. An application may not perform as well with a small cache size, because the cache no longer exceeds the limit.

Storages

The index used by FileStorage was reimplemented using a custom BTrees object. The index maps oids to file offsets, and is kept in memory at all times. The new index uses about 1/4 the memory of the old, dictionary-based index. See the module ZODB.fsIndex for details.

A security flaw was corrected in transactionalUndo(). The transaction ids returned by undoLog() and used for transactionalUndo() contained a file offset. An attacker could construct a pickle with a bogus transaction record in its binary data, deduce the position of the pickle in the file from the undo log, then submit an undo with a bogus file position that caused the pickle to get written as a regular data record. The implementation was fixed so that file offsets are not included in the transaction ids.

Several storages now have an explicit read-only mode. For example, passing the keyword argument read_only=1 to FileStorage will make it read-only. If a write operation is performed on a read-only storage, a ReadOnlyError will be raised.

The storage API was extended with new methods that support the Zope Replication Service (ZRS), a proprietary Zope Corp product. We expect these methods to be generally useful. The methods are:

  • restore(oid, serialno, data, version, transaction)

    Perform a store without doing consistency checks. A client can use this method to provide a new current revision of an object. The serialno argument is the new serialno to use for the object, not the serialno of the previous revision.

  • lastTransaction()

    Returns the transaction id of the last committed transaction.

  • lastSerial(oid)

    Return the current serialno for oid or None.

  • iterator(start=None, stop=None)

    The iterator method isn't new, but the optional start and stop arguments are. These arguments can be used to specify the range of the iterator -- an inclusive range [start, stop].

FileStorage is now more cautious about creating a new file when it believes a file does not exist. This change is a workaround for bug in Python versions upto and including 2.1.3. If the interpreter was builtin without large file support but the platform had it, os.path.exists() would return false for large files. The fix is to try to open the file first, and decide whether to create a new file based on errno.

The undoLog() and undoInfo() methods of FileStorage can run concurrently with other methods. The internal storage lock is released periodically to give other threads a chance to run. This should increase responsiveness of ZEO clients when used with ZEO 2.

New serial numbers are assigned consistently for abortVersion() and commitVersion(). When a version is committed, the non-version data gets a new serial number. When a version is aborted, the serial number for non-version data does not change. This means that the abortVersion() transaction record has the unique property that its transaction id is not the serial number of the data records.

Berkeley Storages

Berkeley storage constructors now take an optional config argument, which is an instance whose attributes can be used to configure such BerkeleyDB policies as an automatic checkpointing interval, lock table sizing, and the log directory. See bsddb3Storage/BerkeleyBase.py for details.

A getSize() method has been added to all Berkeley storages.

Berkeley storages open their environments with the DB_THREAD flag.

Some performance optimizations have been implemented in Full storage, including the addition of a helper C extension when used with Python 2.2. More performance improvements will be added for the ZODB 3.1 final release.

A new experimental Autopack storage was added which keeps only a certain amount of old revision information. The concepts in this storage will be folded into Full and Autopack will likely go away in ZODB 3.1 final. ZODB 3.1 final will also have much improved Minimal and Full storages, which eliminate Berkeley lock exhaustion problems, reduce memory use, and improve performance.

It is recommended that you use BerkeleyDB 4.0.14 and PyBSDDB 3.4.0 with the Berkeley storages. See bsddb3Storage/README.txt for details.

BTrees

BTrees no longer ignore exceptions raised when two keys are compared.

Tim Peters fixed several endcase bugs in the BTrees code. Most importantly, after a mix of inserts and deletes in a BTree or TreeSet, it was possible (but unlikely) for the internal state of the object to become inconsistent. Symptoms then varied; most often this manifested as a mysterious failure to find a key that you knew was present, or that tree.keys() would yield an object that disgreed with the tree about how many keys there were.

If you suspect such a problem, BTrees and TreeSets now support a ._check() method, which does a thorough job of examining the internal tree pointers for consistency. It raises AssertionError if it finds any problems, else returns None. If ._check() raises an exception, the object is damaged, and rebuilding the object is the best solution. All known ways for a BTree or TreeSet object to become internally inconsistent have been repaired.

Other fixes include:

  • Many fixes for range search endcases, including the "range search bug:" If the smallest key S in a bucket in a BTree was deleted, doing a range search on the BTree with S on the high end could claim that the range was empty even when it wasn't.
  • Zope Collector #419: repaired off-by-1 errors and IndexErrors when slicing BTree-based data structures. For example, an_IIBTree.items()[0:0] had length 1 (should be empty) if the tree wasn't empty.
  • The BTree module functions weightedIntersection() and weightedUnion() now treat negative weights as documented. It's hard to explain what their effects were before this fix, as the sign bits were getting confused with an internal distinction between whether the result should be a set or a mapping.

ZEO

For news about ZEO2, see the file ZEO-NEWS.txt.

This version of ZODB ships with two different versions of ZEO. It includes ZEO 2.0 beta 1, the recommended new version. (ZEO 2 will reach final release before ZODB3.) The ZEO 2.0 protocol is not compatible with ZEO 1.0, so we have also included ZEO 1.0 to support people already using ZEO 1.0.

Other features

When a ConflictError is raised, the exception object now has a sensible structure, thanks to a patch from Greg Ward. The exception now uses the following standard attributes: oid, class_name, message, serials. See the ZODB.POSException.ConflictError doc string for details.

It is now easier to customize the registration of persistent objects with a transaction. The low-level persistence mechanism in cPersistence.c registers with the object's jar instead of with the current transaction. The jar (Connection) then registers with the transaction. This redirection would allow specialized Connections to change the default policy on how the transaction manager is selected without hacking the Transaction module.

Empty transactions can be committed without interacting with the storage. It is possible for registration to occur unintentionally and for a persistent object to compensate by making itself as unchanged. When this happens, it's possible to commit a transaction with no modified objects. The change allows such transactions to finish even on a read-only storage.

Two new tools were added to the Tools directory. The analyze.py script, based on a tool by Matt Kromer, prints a summary of space usage in a FileStorage Data.fs. The checkbtrees.py script scans a FileStorage Data.fs. When it finds a BTrees object, it loads the object and calls the _check method. It prints warning messages for any corrupt BTrees objects found.

Documentation

The user's guide included with this release is still woefully out of date.

Other bugs fixed

If an exception occurs inside an _p_deactivate() method, a traceback is printed on stderr. Previous versions of ZODB silently cleared the exception.

ExtensionClass and ZODB now work correctly with a Python debug build.

All C code has been fixed to use a consistent set of functions from the Python memory API. This allows ZODB to be used in conjunction with pymalloc, the default allocator in Python 2.3.

zdaemon, which can be used to run a ZEO server, more clearly reports the exit status of its child processes.

The ZEO server will reinitialize zLOG when it receives a SIGHUP. This allows log file rotation without restarting the server.

What's new in StandaloneZODB 1.0 final?

Release date: 08-Feb-2002

All copyright notices have been updated to reflect the fact that the ZPL 2.0 covers this release.

Added a cleanroom PersistentList.py implementation, which multiply inherits from UserDict and Persistent.

Some improvements in setup.py and test.py for sites that don't have the Berkeley libraries installed.

A new program, zeoup.py was added which simply verifies that a ZEO server is reachable. Also, a new program zeopack.py was added which connects to a ZEO server and packs it.

What's new in StandaloneZODB 1.0 c1?

Release Date: 25-Jan-2002

This was the first public release of the StandaloneZODB from Zope Corporation. Everything's new! :)