JupyterHub Server Start-up Time

[jaesung-son]

Last week at the students' LEAP weekly meeting, we were learning how to use LEAP-Pangeo and mostly all of us experienced long start-up times for the jupyterhub. This is not too big an issue as everyone eventually got onto the server, but it did take most of us 4-5 minutes. We were wondering whether there is a way to reduce the wait and whether IT is aware of this.

[rabernat]

Thanks for this report. These are normal startup times, especially if a lot of people try to log on at the same time. If it is a big problem, we can work with 2i2c to try to mitigate it.

whether IT is aware of this.

We have no "IT" support for LEAP-Pangeo! Just me! 😉

[tz33cu]

We had about 13 people at that time. Is that regarded as "a lot of people"? This info is important for us to know how much simultaneous use of LEAP-Pangeo is reasonable.

[tz33cu]

In addition, the number of node types differs from person to person during our session. Some saw all the types and some only saw "small" and "medium".

[rabernat]

Thanks for these useful questions. I am going to tag @sgibson91 and @choldgraf of 2i2c in case they want to offer some feedback. Here are my responses.

We had about 13 people at that time. Is that regarded as "a lot of people"? This info is important for us to know how much simultaneous use of LEAP-Pangeo is reasonable.

13 people is not a lot of people. The hub can support thousands of simultaneous users. But we must distinguish between capacity and startup time. The total capacity is essentially infinite. But there are tradeoffs involved in startup time. The user notebook sessions are placed within virtual machines. Virtual machines are provisioned on demand. Our costs are directly proportional to the number of virtual machines that are active; therefore, the cluster is configured to scale down to zero when there are no active sessions. When new virtual machines must be provisioned, this takes some time--thus the several minutes you experienced when 13 users all logged on at the same time.

To decrease startup wait times, we could configure the cluster to always keep some extra virtual machines "hot," waiting for a user to come log on. That will translate directly to extra costs. So there are tradeoffs involved. If we feel that the slow startup times are a major problem, we should pursue this. We could also work with 2i2c to try to optimize the startup times in other ways.

In addition, the number of node types differs from person to person during our session. Some saw all the types and some only saw "small" and "medium".

The note types available to different user groups are configured here in the 2i2c infrastructure repo. In order to have access to the large and huge node types, users must belong directly to the leap-pangeo-research team. As we noted in this issue the configuration does not currently support inheritance within teams.

We have also attempted to document these categories in our documentation, but this user access policy is currently in flux as the membership committee has not yet made its recommentation. What is written there is currently inconsistent with the actual configuration.

---

I hope these answers are helpful in explaining the behavior you observed. I deeply appreciate the feedback; it is extremely helpful for us to iteratively improve the service.

[choldgraf]

I agree with all of @rabernat's points above - basically the challenge with startup time is that:

• Nodes can have multiple users on them
• Nodes scale up when we hit our capacity limit on pre-existing nodes
• Starting a session during a node scale-up will take longer, as we wait for the new node
• And anybody that tries to log-in during that time will need to wait as well
• But once the node is ready, start times should be faster until a new scale-up is needed

So the big start time factor is "does a new node need to be created right now?". You can ask for more empty nodes to be sitting there ready for users, but then you are paying for cloud capacity that you're not using.

Note that I've opened up an issue to document this more clearly on the 2i2c service docs as well: 2i2c-org/docs#150