Duplicate IP assignment

[rarenerd]

When deploying multiple VMs in a network with a small amount of IP addresses available, there is a chance that an address gets assigned twice.

The current implementation of acquiring guest IP addresses looks up a list of all available addresses (using a database query) and uses java.util.Random to randomly pick an entry from that list.

cosmic/cosmic-core/server/src/main/java/com/cloud/network/IpAddressManagerImpl.java

Line 1185 in b14fab0

return NetUtils.long2Ip(array[_rand.nextInt(array.length)]);

After this a validation process follows, the NIC object gets populated with various fields and it gets persisted in the database.

cosmic/cosmic-core/engine/orchestration/src/main/java/com/cloud/engine/orchestration/NetworkOrchestrator.java

Line 2085 in a5c6374

vo = _nicDao.persist(vo);

This can take a second or 2 to complete.

With tools like terraform it has become very easy to deploy multiple VMs at the same time, sometimes within milliseconds of each other. So within this window we've seen that the same IP address gets picked for different requests, and it will always happen when there is only a single address left in the pool.