Global Controller spawning agents taking too much time

[Saaketh0]

Global Controller spawning agents is currently sequential, and can be optimized with parallelism.

This happens in the ensure_instances function in ventis/controller/instance_manager.py, on line 33, "for agent_spec in self._agent_specs:"

Since the instance generations are independent of each other, a fix would be to add threading this function, which would speed up the "ventis deploy" command by a factor of the threads able to run on this function.

Main concerns:

• With threads being created simultaneously, host-port assignments will need to be put under a lock to prevent instances from taking the same host-port.
• Too many threads could cause the global controller instance to crash, limit the threads to machine resources.
• Need to make sure failed instances get re-instantiated rather than disappearing.

Link to the function:

canyoncodecore/ventis/controller/instance_manager.py

Line 29 in 62803c2

def ensure_instances(self, agent_specs):

[Saaketh0]

Another potential improvement would be to instantiate the AWS boto3 session once and use it throughout, currently we are starting it for every instance.

Do not know how much of a speedup it is, if it is extremely small just close this issue.