So, first talk is going to be on Dravali Murray Management and Containers by Chantan.
Hi, good morning everyone.
Thank you for coming.
It's a full house.
I can pretend you all here to see me.
And of course, you're actually queueing up two hours early to see Leonard.
I'm just done that, but I'm just going to take that anyway.
It's an honor to be the sort of warm-up act.
Can I just start sort of reiterate what we've said about the mics and stuff.
This also affects what goes on the recording.
So, if you can't hear it and you want to listen back later,
we also still need it pretty quiet.
But there's going to be people coming and going all day long,
so it's quite challenging.
This is the worst room in the whole university for this, I think.
Quick poll.
Can you show a hand, please?
Who here is their first fozz dem?
That's, wow.
Okay, I would say that's probably close to half, actually, which is astonishing.
Please, I hope you enjoy yourselves.
It is quite an experience.
I'm going to do a quick talk about memory tuning of Java for containers.
It's going to be a sort of three-parter.
The first bit, I'm going to talk a little bit about the background in the scene setting,
which Java am I talking about anyway.
Then I'll go into tuning for memory and a bit a little bit on out profiling,
all of that, that's a small chunk.
And then I'll just run over the takeaways.
And hopefully have a little bit of time for questions and answers if there are any.
I've got questions, there's at least one.
I'm going to assume, the assumption I made here is that you are container people,
not really Java people.
And I need to make a confession.
I'm not really a Java person either.
I've been an open source person for a big number of years.
And I've been working on Java on containers for ten years,
but prior to that, I was one of you.
Okay, so I think I'll get this out of the way.
We've all heard of the enterprise programming jokes,
and they're absolutely right.
But there's a conflation going on.
And Java is, in many ways, a unique open source success story,
because it was not a green field open source program.
It was of course close source from the start.
And it was later opened by one of the most
reputationally hostile companies in the world,
whilst it was underpinning billions of dollars of revenue.
And it is now developed by a collaboration of some of the most
egregiously aggressive billion dollar companies in the world
together, and it somehow happens.
So I think it deserves to be recognized.
It's somewhat a unique, oh, a success story there.
Also, we're talking about containers.
This is genuinely some yamble.
I pulled out of some of our stuff I had to do at work.
So let's not throw stones in glass houses.
Okay, so the first bit of which Java is which Java vendor.
Okay, so open JDK is a source distribution.
The open source project produces source,
but it doesn't ship binaries.
That's done by the vendors who are contributing.
And they are all competing with each other
to, first of all, release the builds,
as soon as they can, after the source releases cut.
But also they configure them differently,
some applied different patches.
So where you get your Java from is relevant.
To how you tune it.
So open JDK is the open source project.
Oracle, I tried to understand Oracle's logo trademark license
rules and I couldn't, so their logo doesn't go on slide.
Red type, of course, for another vendor.
Until about six months ago, I was working at Red Hat.
We've now been lifted and shifted into IBM,
and I haven't quite got my head down yet.
So I might occasionally say I'm still at Red Hat.
Just forgive me that.
But I didn't bring my fedora this year.
And Tamarin is a consortium.
It's the Clips Foundation, I think.
Clips Adoptium Tamarin.
That's a very different people contribute to that.
And they produce builds of open JDK for free.
And so if you have no idea where to get it from,
then that's not a bad place to start.
I would say.
Okay.
But then what Java version are we talking about either?
Java 8, JDK 8, anyone who does work on Java,
or experiences it here was probably involved with the Java JDK 8?
I'm afraid to say it's not dead yet.
It's planned to be supported until 2030.
I imagine that I will probably die before it does.
Unfortunately, the Java release model is there's a new feature release
of Java every six months.
Every three years, that release is blessed as a long-term support release.
And every quarter there's a roll-up patch release for the company's supported versions.
So that's the more full CV fixes in it.
So another one you may have experienced JDK 11 is now pretty long in the tooth.
That's been around quite a long time.
And actually, that's dying this year, apparently.
Until the 11th hour when a very rich customer's
tells to keep supporting it in which case that number might change.
But there are subsequent LTS versions that would be better choices
if you're doing new deployments now.
25 came out this year.
Well, just a year gone coincidentally in 2025.
It would be a better choice.
And this is relevant for container memory tuning,
because we are introducing new features and improvements
in these Java versions that are going forward
that you're not getting if you're stuck back on eight.
So Java is a managed language, or the J.
I mean, yeah.
One of these things is caveats.
I'm talking about Java.
The managed language is a garbage collector.
There's a runtime.
It does jit.
But another way to manage memory would be to do something
a bit different, like compile it in native.
Oracle Labs, who are separate part of Oracle from the Java team
Oracle, have a product called Graal VM, which is a native
compiler for Java.
You can use that to build native binaries.
And then you don't have the jit.
You don't have hotspot.
That's quite one way you can use it is to frame work like
Quarkus, which is a sort of batteries included,
Java development framework, which supports
doing native builds out of the box.
That's pretty much all I'm going to say on native builds
and stuff, because mostly I'm talking about the J.V.
I'm the rest of this talk.
All right.
The J.V.M. is container aware, in the sense it reads memory limits
that have been set via C groups.
Either version one or version two.
It's so good we did it twice.
Old documentation might mention this flag,
which in the early days was necessary for Java to actually
on a C group values.
It's not needed anymore.
We have it's just defaults to on.
You can turn it off if you really want to.
And we've backported C group to be one and V2 all the way back
to JDK8.
So wherever Java you're using, it can read C group memory limits.
It has what we call container awareness.
So a brief overview of what Java memory,
which I'm just view of memory is.
The lion's share of the memory that Java manages is the heap.
It's an object oriented language.
Creates a lot of objects.
There was an allocated on the heap.
The garbage collector operates on the heap.
This is not proportional exactly.
It's basically the biggest chunk of RAM that Java is going to use.
There's a little bit called metaspace, which is separate.
So various bits of class metadata, internal stuff,
static variables are managed here.
That's utterly independent from the heap.
This doesn't get garbage collected by the Java garbage collector.
The management of metaspace memory interacts with the garbage collector.
In complicated ways that I'm not going to so expand on here.
But it's a smaller chunk of RAM memory than a heap.
There's even more that unfortunately.
So of course the stacks that the actual Java threads have stacks.
There's for completion sake.
This program count is for managing which line which instruction is being executed
in each of those Java threads.
The JVM is a native program at the end of the day.
So there's also native stacks when per OS thread.
And it's possible to allocate native memory as well on the heap.
So that's not managed by the Java garbage collector.
But it can be used by native libraries are linked in via various methods.
Or there's netty which is a popular web tool uses a lot of this.
And it can be hard to reason about.
So when it comes to the JVM and trying to decide how to best utilize the available memory it has.
This is the picture it has.
And some of that, even though it's aware of it, it's without sign as control.
But in a container context, we also have potentially sub-processes that we have created.
Which have their own RAM.
If you're talking about Kubernetes or something like that, then you're going to have probes.
Liveness probes, readiness probes.
They are going to execute within the context of the container and therefore under the subject to that.
Whatever the memory limit is.
There's also.
If a DevOps person just bonds a shell to see what's going on, that needs to be accounted for as well.
And the takeaway here really is that the JVM does not have an overview of full and complete overview of the RAM requirements that are going to happen.
Inside a container, even if it is ostensibly the main thing running.
So when you create a container with a Java payload and you give it X amount of gigs of RAM, it has to allow for the fact that there's going to be other bits and bombs going on.
And it's hard for it to predict what those are going to need.
Okay.
So the biggest widget you can tweak in the JVM for controlling the use of memory is to set a limit on how much.
How big the heap can be.
And the JVM defaults to 25% of available memory, which is a pretty low number.
So you give two gigabytes of RAM to Java payload and do nothing else.
It won't allocate more than half a gig of that for heap.
That's because of this headroom problem.
And it's also because it has to ask are we in a container? What is a container?
It's a leaky abstraction, having a C group imposed memory limit does not necessarily mean we are a container.
And the consequences of allocating a much larger proportion of the available memory in a different context, like a multi-user context or something, could be pretty catastrophic.
So the JVM defaults to a really low value for this.
In reds out we move this default up to 80% which seems to be a slightly more sensible default if the majority of the work is actually going to be the JVM.
And the majority of it's RAM is going to be heap.
Again, we've left 20% there for a headroom.
But ultimately it's very limited in terms of what we can do without better understanding of what your app is actually doing.
And only you have that.
Okay.
The average collection, Java has a lot of garbage collectors.
I've put five on this slide, that's not all of them.
In the near future, the default garbage collector in all circumstances will be what's called the G11 which is a balance one.
Otherwise, they broadly organised into throughput-oriented garbage collectors.
So those ones minimise them out of time spent doing garbage collection, work versus application work, or latency-oriented,
where the important thing is how fast the application can respond to events.
Those two things are somewhat in competition with each other.
So if you have an application which needs to be very responsive, then you want to be going for one of these latency-based ones.
Otherwise, a throughput-one perhaps.
On the latency side of things, there's a garbage collector called Shen and Doer, which was added in,
I have to be the big slide, 12 technically.
But, and this is, again, why it's important to understand which vendor you're using, Oracle do not turn it on.
So when they build Java and distribute Java, they disable building the Shen and Doer garbage collector and do not distribute it with their releases.
They consequently also wrote ZGC, which landed later, which they do switch on.
So what's available to you might depend on what your vendors decisions are.
Write out backported Shen and Doer to eight, but not upstream.
So that's in the Red Hat builds and not in open JDK's source.
Another GC, this might actually be the last one, but this one's a really strange one.
It basically doesn't do anything.
So it manages memory allocation for the heap, but it never frees anything.
Which means it doesn't spend any time freeing anything.
And if you run out of RAM, you hit an out memory error, it your program is terminated.
It was originally introduced probably as a sort of exercise in testing the GUB collection API to say, look, this is how easy it is to write a GUB collector.
But it's potentially useful if you have an application where you are very carefully managing how many allocations there are.
So you know how much you're using or you need such fast response time that you cannot afford to spend any time with garbage collection at all.
Or if you've got an out of memory scenario, it is better for Java to die and for your scheduler to manage that and to rewrite traffic and do high availability stuff.
If it perhaps is doing functions in a service, for example, that might be useful garbage collector for you.
There's a talk about that first in 19.
I've put all the links to talk to stuff in the speaker notes.
So if anything's of interest, if I mention anything, it's not written on the slide. It'll be in the PDF when I eventually stick it on on the website.
So take away from all that is, use a recent version of Java if you've portable account and here are a few reasons why.
So in 2016 or 2021, some work was done on improving the way that Metispace memory was managed.
It's still absolutely independent of the heap, so it's not garbage collected, but this used an allocation management scheme.
A little bit similar to one of the Linux colonel users and massively improved the performance of the Metispace, which in some pathological situations can actually be a big problem.
This command line argument here was added at the time to allow you to tune its behavior between balanced or aggressive.
I believe more recently that switch has gone and it's just on all the time in balanced mode.
But you don't get the benefit of that unless you're running at least JDK-16.
This is relatively new, and I think it's ergonomically a little, it's going to improve, but there's a lot of work that happens when Java application start,
which it could be better off cached and just read back from disk, so that's class initialization and things like that.
So this work here is essentially a head of time compiling some of that.
You need to, it's a three step process at the moment. You run your application, first of all, with some command line flags from the first step to a quarter profile of what classes are loaded and what things need to be cached.
The second command then is independent at the moment. I think it will probably be merged with the first one in due course.
The second one essentially translates that into the cache.
And the third time, the third step, run your application again this time using the cache.
This can be tremendous for improving the start of time, which are perhaps because typically they run relatively slowly until the total enough hot path analysis is taking place.
And the jet kicks in and then it starts doing negative compilation. So this helps to accelerate that initial step.
Another piece of work was reducing the size of object headers, so typical Java work loads allocate a lot of very small objects.
So the actual sort of per object footprint turns out to be very important.
This landed, yeah, last year, and you present you need to switch it on, but I think it's reasonable to assume that over time this will become a default as well.
And apparently for some of the instrumentation they've done this can reduce memory utilization by 20% essentially for free.
Okay. I have a little bit on application profiling so that the takeaway of what I've said so far I think is that the JVM is improving.
And there are various things you can do or have an automatically to reduce memory pressure.
But it has to make default assumptions based on work loads not just container workloads and that limits to what we can actually achieve with auto tuning.
So to get the absolute best performance out of Java in a container context, you need to profile your application just like outside a container context.
Outside of container context, the tools that you might wish to use are something called Java Flight Recorder, which effectively captures metrics about a running application.
And Java Mission Control is this GUI app that I've screened to actually browse a capture and to look at and see what's going on.
And the sort of tuning wizards can use that information to figure out what you should try and do differently to really squeeze that performance out of your app.
This is a little awkward to use in a container context though of course.
So my colleagues formerly operator have worked on a tool called Cryo Start. This would be a really good place for a live demo.
But sorry, did you make them a screenshot?
Cryo Start effectively is some tooling and to sort containerize the collection and presentation of Java Flight Recorder data in a container context.
So it sits nicely next to your open shift deployment or Kubernetes deployment and manages the security of capturing those traces and providing access to the relevant to them to the relevant people and dashboarding analytics.
So that's the approach I would use if you wanted to do application profiling of Java workloads and containers.
Okay, so running towards the end of my time.
Take away, keep up to date with JDK, get off eight if you've taught can and make take advantage of the new features that are arriving every six months.
Over auto tuning is improving, things are going to get better for free, but this is going to take you so far and up profiling remains important for squeezing the best performance out of your Java app as it does for any other payload.
So that's it. Thank you.
All right, we've got about one minute for question, just a note before we do it out.
When you those tools in front of the room, if you need to exit, you can exit through the two doors here instead of going all the way back and downstairs.
That tends to be a bit easier to leave the room, but that's exit on this, you can come back that way.
Okay, any question? All right, let's do the easy if front one first.
Where compact object cutter is not turned on by default, sorry, 25 and onwards?
That's a very good question. I think it's because generally speaking, Java features are added very conservatively and although the developers who proposed this and advocate for it have perform metrics to justify its inclusion.
It's an easier, it's easier to get it into the main line, JVM, if it's not on by default, sort of day one.
And like, follow up with, would say, look, it's been in for six months, people are using it, there's been no problems, it's time to make it a default.
Sorry? It could be on that default later, yes.
Okay, my question is, with this ahead of time cashing, is it possible to fill the cash outside the container and then deliver the container with the cash already in place?
Yes.