CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

1.2.0 - 2018-12-20

This version contains 155 contributions from 11 contributors. There are 113 files changed; 19,085 insertions; and 4,323 deletions.

Added

• Cancel queries on Context.Done() (#1773)
• Union In Place (#1766, #1774)
• Import benchmarking (#1771)
• Add GroupBy() Filter (#1753)
• Add /internal/translate/keys endpoint (#1751)
• CircleCI: Add race detector to parallel build, default to Go 1.11. (#1756)
• Add distributed tracing. (#1684)
• Add NoStandardView field option (#1733)
• Add some stat tracking to roaring implementation (#1743)
• Add cluster fault testing using docker-compose and pumba (#1717)
• Allow backslash, carriage return in PQL strings (#1713)
• Add base system, curl and jq for debug and checks (#1707)
• Add Rows and GroupBy functionality (#1647)
• Add clear functional option for imports (#1699)
• Implement tracking of available shards to help support sparse datasets (#1600, #1695, #1624, #1663)
• Add missing rowID/Key columnID/Key tests (#1683)
• Add Store() operation to PQL (#1666)
• Add diagnostics CPUArch field (#1671)
• Add CircleCI step to generate Docker image and push to Docker hub (#1673)
• Implement ClearRow() query (#1645)
• Add support for Bool fields (#1658)
• Make translate map size configurable (#1653)
• Add DirectAdd function to roaring.Bitmap (#1646)
• Implement Roaring import (#1622, #1738)
• Add Not() query (#1635)
• Implement Options call and excludeRowAttrs, excludeColumns, columnAttrs and shards args (#1631)
• Add field options to pilosa import (#1625)
• Implement column existence tracking (#1788, #1672, #1628)

Changed

• Convert the anti-entropy logic to use ImportRoaring instead of QueryNode (#1780)
• Simplify require-* logic in Makefile (#1755)
• Cleanup logging (#1748)
• Remove TravisCI, add CircleCI shield (#1740)
• Upgrade Peg dependency and regenerate grammar (#1725)
• Upgrade to protoc 3.6.1 (also updated protoc-gen-gofast) (#1724)
• Move column attrs logic to executor (#1677)
• Shrink container bit count to int32 (#1664)

Performance

• Remove bounds check (#1619)
• Improve benchmarking and performance (#1741)

Fixed

• Ensure internal client closes all response bodies (#1795)
• Allow translate log entry buffer to grow (#1787)
• Add Gopkg.lock as a dependency for vendor target (#1790)
• Cluster resize fix (#1785)
• Attempt to fix deadlock by releasing view lock before broadcasting (#1782)
• Fix bug where cluster goes into RESIZING instead of NORMAL (#1777)
• Propogate updates to node details (not just additions and deletions) (#1769)
• Fix arm64 support (#1764)
• Fix data races (#1750)
• Fix fragment checksums race condition (#1749)
• Import cmd field type flag (#1732)
• Increase the translate file size for tests/benchmarks (#1744)
• Prevent panic in Bitmap.UnmarshalBinary when there is no data (#1742)
• Remove unused rule from peg grammar (#1737)
• Improve Internal Client errors (#1729)
• Forward imports to non-coordinator shards (#1719)
• Fix double escapes in PQL grammar (#1727)
• Ensure btree comparison doesn't fail for smallish N (#1712)
• Drop now-superfluous methodNotAllowedHandler (#1711)
• Use pilosa.Logger everywhere (#1674)
• Ensure view closes fragment on broadcast error (#1675)
• Prevent closing os.Stderr (used in verbose test logging) (#1696)
• Allow holder to close/open/close without panic on closing closed channel (#1686)
• Fix bug with Range() queries with field keys (#1679)
• Sync query validation for handlers (#1676)
• Wrap translation store errors, decrease test map size to prevent failure on 32-bit (#1665)
• Fix pass-by-value issue in proto decode (#1662)
• Do not run prerelease in CI if this is a pull request (#1655)
• Ensure mutex imports unset previous columns (#1656)
• Treat import timestamps as UTC (#1651)
• Remove unused log buffers from test cluster, fixes race (#1612)
• Add --field-keys and --index-keys options to pilosa import (#1621)
• Use passed stdin, stdout, and stderr in the cmd package (#1620)
• Update Go client sample to match latest master (#1614)

1.1.0 - 2018-08-21

This version contains 32 contributions from 5 contributors. There are 89 files changed; 2,752 insertions; and 1,013 deletions.

Added

• Add CircleCI (#1610)
• Add key translation to exports (#1608)
• Support importing key values (#1599, #1601)
• Treat coordinator as primary translate store (#1582)
• Add DEGRADED cluster state and handle gossip NodeLeave events correctly (#1584)
• Add linters to gometalinter and fix related issues (#1544, #1543, #1540, #1539, #1537, #1536, #1535, #1534, #1530, #1529, #1528, #1526, #1527)
• Add mutex field type (#1524)
• Fragment rows() and rowsForColumn() (#1532)

Fixed

• Fix race on replicationClosing channel (#1607)
• Prevent anti-entropy and cluster resize from running simultaneously (#1586)
• Require a valid port that isn't greater than 65,535 (#1603)
• Add view parameter to sync logic for syncing time fields (#1602)
• Fix translator in cluster environment (#1552)
• Use string prefix instead of equality so json error message will pass on all Go versions (#1558)

[1.0.2] - 2018-08-01

This version contains 11 contributions from 3 contributors. There are 30 files changed; 1,569 insertions; and 1,215 deletions.

Fixed

• Fix documentation (#1503, #1495, #1551)
• Fix places where empty IndexOptions were being used (#1547)
• Fix translator syncing bug in cluster environments (#1552)
• Fix race condition in translate_test (#1541)
• Add IndexOptions to IndexInfo json response (#1542)
• Add proper locking to cluster code to prevent races (#1533)
• Re-export erroneously unexported func Row.Intersect (#1502)
• Update parser to handle row keys on SetRowAttrs() (#1555)

[1.0.1] - 2018-07-11

This version contains 12 contributions from 4 contributors. There are 11 files changed; 133 insertions; and 39 deletions.

Fixed

• Use dep ensure -vendor-only for build repeatability (#1491)
• Make sure time range views are calculated correctly across months (#1485)
• Fix up error handling, add a configurable timeout to http handler closing (#1486)
• Add gossip Closer (#1483)
• Update docs references to WebUI naming (console) and installation (#1493)

1.0.0 - 2018-07-09

This version contains 218 contributions from 7 contributors. There are 184 files changed; 21,769 insertions; and 20,275 deletions.

Added

• ID-Key Translation (#1337)
• Add CORS support to handler (#1327)

Changed

• HTTP handler updates (#1408, #1399, #1441, #1375, #1433, #1444, #1388, #1309, #1302, #1304, #1465, #1466)
• Refactor/improve tests (#1437, #1434, #1435, #1425, #1418, #1419, #1413, #1394, #1387, #1386, #1378, #1364, #1348, #1340, #1297)
• Simplify inter-node communication (#1428, #1427, #1412, #1398, #1391, #1389)
• Make gossip's interface to Pilosa the API struct (#1452)
• Rename slice to shard (#1426)
• Clearbit for time fields (#1424)
• Update docs (#1390, #1329, #1305, #1296, #1461)
• Simplify server setup (#1417, #1393,#1451)
• Refactor API (#1407)
• Rewrite PQL parser and add various improvements/simplifications (#1382, #1402, #1354, #1463)
• Rename "frame" to "field" (#1395, #1362, #1360, #1358, #1357, #1355)
• Optimize count (#1365)
• Simplify bitmap max function (#1333)
• Rename "bit" to "column" for clarity (#1326)
• Rename pilosa.Bitmap to Row (#1311)
• Invert encoding/decoding and remove internal references (#1454)

Removed

• Rename (unexport) many items to reduce public API footprint prior to 1.0 release (#1470, #1458, #1450, #1449, #1448, #1447, #1446, #1438, #1443, #1440, #1439, #1409, #1392, #1374, #1372, #1369, #1367, #1366, #1351, #1420, #1416, #1397)
• Remove dead code (#1432, #1457, #1421, #1411, #1377, #1393, #1462)
• Remove view argument from Field.SetBit and Field.ClearBit (#1396)
• Remove WebUI (now contained in a separate package) (#1363)
• Remove bench command (#1347)
• Remove "view" from API, handler, docs (#1346)
• Remove backup/restore stuff (#1339, #1341)
• Remove inverse frame functionality (#1335)
• Remove rangeEnabled option (#1332)
• Remove index and field MarshalJSON (#1468)

Fixed

• Fix a few data races (#1423)
• Fix for crash while removing containers (#1401)
• Allow dashes in frame names (#1415)
• Fix generate-config command, use single toml lib (#1350)

0.10.0 - 2018-05-15

This version contains 93 contributions from 8 contributors. There are 93 files changed; 4,495 insertions; and 5,392 deletions.

Added

• Add B+Tree containers (Enterprise Edition) (#1285)
• Add /info endpoint (#1236)

Changed

• Wrap errors (#1271, #1258, #1274, #1270, #1273, #1272, #1260, #1259, #1256, #1257, #1261, #1262, #1263, #1265)

Removed

• Remove unused code (#1286)
• Remove input definition, add install-stringer to Makefile (#1284)
• Remove /id and /hosts endpoints. Add local ID to /status (#1238)
• Remove API.URI (#1255)

Fixed

• Assorted docs fixes (#1281, #1269)
• Update PQL syntax in bench subcommand (#1279)
• Update help menu in WebUI (#1278)
• Fix dead lock (#1268)
• Make sure gossipMemberSet.Logger is set during server setup (#1266)
• Make sure ~ is expanded in NewServer; BroadcastReceiver uses temp path (#1242)
• Avoid creating a slice of nil timestamps on Import() (#1234)
• Fixup internal client (#1253)

0.9.0 - 2018-05-04

This version contains 188 contributions from 12 contributors. There are 141 files changed; 17,832 insertions; and 7,503 deletions.

Please see special upgrading instructions for this release.

Added

• Add ability to dynamically resize clusters (#982, #946, #929, #927, #917, #913, #912, #908)
• Update docs to include cluster-resize config and instructions (#1088)
• Add support for lists of gossip seeds for redundancy (#1133)
• Add HTTP Handler validation (#1140, #1121)
• Add validation around node-remove conditions (#1138)
• broadcast.SendSync field creation and deletion to all nodes (#1132)
• Spread recalculate caches to all nodes. Fixes #1069 (#1109)
• Add QueryResult.Type to protobuf message to distiguish results at the client (#1064)
• Modify pilosa import to support string rows/columns (#1063)
• Add some statsd calls to HolderSyncer (#1048)
• Add support for memberlist gossip configuration via pilosa.Config (#1014)
• Add local and cluster IDs (#1013, #1245)
• Add HolderCleaner and view.DeleteFragment (#985)
• Add set-coordinator endpoint (#963)
• Implement Min/Max BSI queries (#1191)
• Log time/version to startup log (#1246)
• Documentation improvements (#1135, #1154, #1091, #1108, #1087, #1086, #1026, #1022, #1007, #981, #901, #972, #1215, #1213, #1224, #1250)

Changed

• Put Statik behind an interface (#1163)
• Refactor diagnostics, inject gopsutil dependency (#1166)
• Use boolean instead of address to configure coordinator (#1158)
• Put GCNotify behind an interface (#1148)
• Replace custom assembly bit functions with standard go (#797)
• Improve roaring tests (#1115)
• Change configuration cluster.type (string) to cluster.disabled (bool) (#1099)
• Use NodeID instead of URI for node identification (#1077)
• Change gossip config from DefaultLocalConfig to DefaultWANConfig (#1032)
• Use binary search in runAdd (#1027)
• Use HTTP handler for gossip SendSync (#1001)
• Group the write operations in syncBlock by MaxWritesPerRequest (#950)
• Refactor HTTPClient handling (#991)
• Remove FrameSchema. Move Fields to the Frame struct (#907)
• Refactor pilosa/server (#1220)
• Clean up flipBitmap and add tests (#1223)
• Move pilosa.Config to pilosa/server.Config (#1216)
• Vendor github.com/golang/groupcache/lru (#1221)

Removed

• Remove the Gossip stutter from memberlist-related config options (#1171)
• Remove old GossipPort and GossipSeed config options (#1142)
• Remove cluster type http from docs (#1130)
• Remove holder.Peek, combine with HasData, move server logic (#1226)
• Remove PATCH frame endpoint (#1222)
• Remove Index.MergeSchemas() method (#1219)
• Remove references to Input Definition from the docs (#1212)
• Remove Index.TimeQuantum (#1209)
• Remove SecurityManager. Implement api restrictions in api package. (#1207)

Fixed

• Handle the scheme correctly in config.Bind (#1143)
• Prevent excessive sendSync (createView) messages. (#1139)
• Fix a shift logic bug in bitmapZeroRange (#1110)
• Fix node id validation on set-coordinator (#1102)
• Avoid overflow bug in differenceRunArray (#1105)
• Fix bug in NewServerCluster where each host was its own coordinator (#1101)
• Fix count/bitmap mismatch bug (#1084)
• Fix edge case with Range() calls outside field Min/Max. Fixes #876. (#979)
• Bind the handler to all interfaces (0.0.0.0) in Dockerfile. Fixes #977. (#980)
• Fix nil client bug in monitorAntiEntropy (and test) (#1233)
• Fix crash due to server.diagnostics.server not set (#1229)
• Fix some cluster race conditions (#1228)

Deprecated

• Deprecate RangeEnabled option (#1205)

Performance

• Add benchmark for various container usage patterns (#1017)

[0.8.8] - 2018-02-19

This version contains 1 contribution from 2 contributors. There are 4 files changed; 1,153 insertions; and 618 deletions.

Fixed

• Bug fixes and improved test coverage in roaring (#1118)

[0.8.7] - 2018-02-12

This version contains 1 contribution from 1 contributors. There are 2 files changed; 84 insertions; and 4 deletions.

Fixed

• Fix a shift logic bug in bitmapZeroRange (#1111)

[0.8.6] - 2018-02-09

This version contains 2 contributions from 2 contributors. There are 3 files changed; 171 insertions; and 6 deletions.

Fixed

• Fix overflow bug in differenceRunArray #1106
• Fix bug where count and bitmap queries could return different numbers #1083

[0.8.5] - 2018-01-18

This version contains 1 contribution from 1 contributor. There is 1 file changed; 1 insertion, and 0 deletions.

Fixed

• Bind Docker container on all interfaces (#1061)

[0.8.4] - 2018-01-10

This version contains 4 contributions from 3 contributors. There are 17 files changed; 974 insertions; and 221 deletions.

Fixed

• Group the write operations in syncBlock by MaxWritesPerRequest (#1038)
• Change gossip config from memberlist.DefaultLocalConfig to memberlist.DefaultWANConfig (#1033)

Performance

• Change AttrBlock handler calls to support protobuf instead of json (#1046)
• Use RLock instead of Lock in a few places (#1042)

[0.8.3] - 2017-12-12

This version contains 1 contribution from 1 contributor. There are 2 files changed; 59 insertions; and 42 deletions.

Fixed

• Protect against accessing pointers to memory which was unmapped (#1000)

[0.8.2] - 2017-12-05

This version contains 1 contribution from 1 contributor. There are 15 files changed; 127 insertions; and 98 deletions.

Fixed

• Modify initialization of HTTP client so only one instance is created (#994)

[0.8.1] - 2017-11-15

This version contains 2 contributions from 2 contributors. There are 4 files changed; 27 insertions; and 14 deletions.

Fixed

• Fix CountOpenFiles() fatal crash (#969)
• Fix version check when local is greater than pilosa.com (#968)

0.8.0 - 2017-11-15

This version contains 31 contributions from 8 contributors. There are 84 files changed; 3,732 insertions; and 1,428 deletions.

Added

• Diagnostics (#895)
• Add docker-build make target for repeatable Docker-based builds (#933)
• Add documentation on importing field values; fixes #924 (#938)
• Add flag documentation and tests, remove "plugins.path" (#942)
• Add TLS support (#867)
• Add TLS cluster how to (#898)
• Add support for gossip encryption (#889)
• Add Recalculate Caches endpoint (#881)
• Add search-friendly documentation for BSI range query syntax (#955)

Changed

• Remove unneeded Gopkg.toml constraints and update all dependencies (#943)
• Remove row and column labels in webUI (#884)
• Internal Client refactoring (#892)
• Remove column/row labels for input definition (#945)
• Update dependencies and Go version (#878)

Fixed

• Skip permissions test when run as root. Fixes #940 (#941)
• Address "connection reset" issues in client (#934)
• Fix field value import: Use signed int and respect field minimum (#919)
• Constrain BoltDB to version rather than specific revision (#887)
• Fix bug in environment variable format (#882)
• Fix overflow in differenceRunBitmap (#949)

Performance

• Use FieldNotNull to improve efficiency of BETWEEN queries (#874)

[0.7.2] - 2017-11-15

This version contains 1 contribution from 1 contributor. There is 1 file changed; 16 insertions; and 1 deletion.

Changed

• Bump HTTP client's MaxIdleConns and MaxIdleConnsPerHost (#920)

[0.7.1] - 2017-10-09

This version contains 3 contributions from 3 contributors. There are 14 files changed; 221 insertions; and 52 deletions.

Changed

• Update dependencies and Go version (#878)

Performance

• Leverage not-null field to make BETWEEN queries more efficient (#874)

0.7.0 - 2017-10-03

This version contains 59 contributions from 9 contributors. There are 61 files changed; 5207 insertions; and 1054 deletions.

Added

• Add HTTP API for fields (#811, #856)
• Add HTTP API for delete views (#785)
• Modify import endpoint to handle BSI field values (#840)
• Add field Range() support to Executor (#791)
• Support PQL Range() queries for fields (#755)
• Add Sum() field query (#778)
• Add documentation for BSI (#861)
• Add BETWEEN for Range queries (#847)
• Add Xor support for PQL (#789)
• Enable auto-creating the schema on imports (#837)
• Update client library docs (#831)
• Handle SIGTERM signal (#830)
• Add cluster config example to docs (#806)
• Add ability to exclude attributes and bits in Bitmap queries (#783)

Fixed

• Fix panic when iterating over an empty run container (#860)
• Fix row id zero bug (#814)
• Fix cache invalidation bug (#795)
• Set container.n in differenceRunRun (#794)
• Fix infinite loop in bitmap-to-array conversion (#779)
• Fix CountRange bug (#773)

Deprecated

• Remove support for row/column labels (#839)

Performance

• Refactor differenceRunArray (#859)
• Update fragment.FieldSum to use roaring IntersectionCount() (#841)
• Add roaring optimizations (#842)
• Convert lock to read lock (#848)
• Reduce Lock calls in executor (#846)
• Implement container.flipBitmap() to improve differenceRunBitmap() (#849)
• Reuse container storage on UnmarshalBinary to improve memory utilization (#820)
• Improve WriteTo performance (#812)

0.6.0 - 2017-08-11

This version contains 14 contributions from 5 contributors. There are 28 files changed; 4,936 insertions; and 692 deletions.

Added

• Add Run-length Encoding (#758)

Changed

• Make gossip the default broadcast type (#750)

Fixed

• Fix CountRange (#759)
• Fix differenceArrayRun logic (#674)

0.5.0 - 2017-08-02

This version contains 65 contributions from 8 contributors (including 1 volunteer contributor). There are 79 files changed; 7,972 insertions; and 2,800 deletions.

Added

• Set open file limit during Pilosa startup (#748)
• Add Input Definition (#646)
• Add cache type: None (#745)
• Add panic recovery in top level HTTP handler (#741)
• Count open file handles as a StatsD metric (#636)
• Add coverage tools to Makefile (#635)
• Add Holder test coverage (#629)
• Add runtime memory metrics (#600)
• Add sorting flag to import command (#606)
• Add PQL support for field values (WIP) (#721)
• Set and retrieve field values (WIP) (#702)
• Add BSI range-encoding schema support (WIP) (#670)

Changed

• Move InternalPort config option to top-level (#747)
• Switch from glide to dep for dependency management (#744)
• Remove QueryRequest.Quantum since it is no longer used (#699)
• Refactor test utilities into importable package (#675)

Fixed

• Add mutex for attribute cache (#729)
• Use log-path flag to specify log file (#678)

0.4.0 - 2017-06-08

This version contains 53 contributions from 13 contributors (including 4 volunteer contributors). There are 96 files changed; 6373 insertions; and 770 deletions.

Note that data files created in Pilosa < 0.4.0 are not compatible with Pilosa 0.4.0 as a result of #520.

Added

• Support metric reporting through StatsD protocol (#468, #568, #580)
• Improve test coverage for ctl package (#586)
• Add support for bit flip (negate) in roaring (#592)
• Add xor support to roaring (#571)
• Improve WebUI autocomplete (#560)
• Add syntax hints tooltip to WebUI (#537)
• Implement 'config' CLI command (#541)
• Move docs into repo (#563)
• Add inverse TopN() support (#551)
• Add various Makefile updates (#540)
• Provide details on Glide checksum mismatch (#546)
• Add Docker multi-stage build (#535)
• Support inverse Range() queries (#533)
• Support colon commands in WebUI (#529, #510)

Changed

• Increase default partition count from 16 to 256 (BREAKING CHANGE) (#520)
• Validate unknown query params (#578)
• Validate configuration file (#573)
• Change default cache type to ranked (#524)
• Add max-writes-per-requests limit (#525)

Fixed

• Add "make test" to PHONY section of Makefile (#605)
• Fix failing tests when IPv6 is disabled (#594)
• Add minor docs fix, indent in JSON (#599)
• Fix BroadcastHandler handle missing index error (#597)
• Add WebUI fixes (#589)
• Fix support for 32-bit Linux (#549, #565)
• Fix 3 separate bugs in bitmapCountRange (#559)
• Add client support for MaxInverseSliceByIndex (#555)
• Fix bug in handleGetSliceMax (#554)
• Default to standard view in export command (#548)
• Fix vet issues with the assembly code in Roaring (#528)
• Prevent row labels that match the column label (#503)
• Fix roaring test: TestBitmap_Quick_Array1 (#507)
• Don't try to create inverse views on Import() when inverseEnabled is false (#462)

Performance

• Set n based on array length instead of incrementing repeatedly (#590)
• Rewrite intersectCountArrayBitmap for perf test (#577)
• Check for duplicate attributes under read lock on insert (#562)