Ironic Bare Metal RSS Feed

No title

A Page

Our Project is a global open source community independently governed at the OpenStack Foundation. Our ethos is user-driven and our culture is welcoming and respectful. We invite you to try out Our Project, provide your feedback and get involved in contributing to the code.

Try Our Project

Whitepaper: https://yourproject.org/collateral/White_Paper.pdf - an overview of the design approach that Airship 2.0 is using to declaratively manage open infrastructure
Git Repository: https://opendev.org/yourproject/

Contribute

Developer Guide: https://yourproject.org/readthedocs/en/latest/dev-getting-started.html

Collaborate

Meet the team and collaborate with us at OpenDev + PTG.

Learn

Project Update at the Denver Open Infrastructure Summit, April 2019

Get Started in Development, April 2022

More to Learn

Your Project Update at the Berlin Open Infrastructure Summit, November 2018

Realizing configuration management reliably and predictably, November 2018

Making lifecycle management for open infrastructure repeatable and predictable, May 2018

Wiki

Community documentation and other resources are available at wiki.openstack.org/wiki/yourproject

Standing Meetings

The Our Project community has a very active schedule of regular meetings. Details are available here.

Communications

OFTC IRC: #ourproject
Mailing Lists: lists.yourproject.org
E-mail: mailto:info@yourproject.org
Twitter: @yourproject

Governance

Our Project is governed according to the “four opens”:

Open Source
Open Design
Open Development
Open Community

Technical decisions will be made by technical contributors and a representative technical leadership committee. The community is committed to diversity, openness, encouraging new contributors and leaders to rise up.

Code of Conduct

Our community follows the OpenStack Foundation Code of Conduct.

Empowering End-Users in Ironic 35.0

Tue, 17 Mar 2026 12:00:00 GMT

Even when only considering its integration with OpenStack, Ironic has multiple different customers at any given time. The admin who installs and manages the conductors. The project managers who might online hardware and configure nodes in Ironic. The end-users who provision bare metal instances through Nova's compute API. In Ironic 35.0 (OpenStack 2026.1, Gazpacho), we made two major feature improvements with one of those users in mind in particular: the end-user, by empowering them with more options about how to configure their instances at deploy time.

Autodetect Deploy Interface

The first of these user-facing features is our new autodetect deploy interface. This permits changing the selected deployment method based on image metadata. Previously, this was selected by node configuration – by either an admin or a manager – and left entirely out of the hands of the end-user. Now, by configuring nodes with the autodetect deploy interface, we can enable end users to select what deployment method they would like by choosing what image to deploy.

Detection is based off image metadata. The three interfaces that we can support autodetection for today are bootc, ramdisk, and direct:

bootc is selected when an OCI container is provided as the deployment image, and that OCI container does not contain a full disk image. This detection method will never trigger for Nova end-users, as OCI container urls are not supported for Nova deployment.
ramdisk is selected when a Glance image is provided with ironic_ramdisk=True in the metadata. The autodetect interface then looks for additional metadata pointing to a boot iso or kernel/ramdisk pairing for boot.
direct is the fallback deployment interface used when none of the others match.

If your nodes are configured today for direct deployment, you can likely switch them to autodetect with no disruption.

Please see the autodetect deploy interface documentation for more specific information on how to configure this feature.

Trait-based port scheduling

The second of these user-facing features is subtle, but powerful. Previously, when provisioning nodes with a fully integrated OpenStack, operators had few options as to how to map networks to ports or portgroups – a simple match on physical_network. This is functional; but static. Instead, we wanted to give end-users more flexibility in how their virtual interfaces from Neutron are mapped into a physical machine.

First, a short aside: what is a trait? Traits are a way for Nova to track machines configured in a certain way. For instance, if you configure a Nova flavor to have the property trait:HW_CPU_X86_AVX2=required, virtual instances booted with that flavor will be placed onto a hypervisor with AVX2-supporting CPUs. In a similar way, Ironic uses custom traits to advertise different ways that Ironic can configure a node to fulfill a flavor request. Trait-based port scheduling is based on this concept: an admin can configure a flavor to request a specific trait from Ironic which will inform Ironic how you want your ports scheduled.

Configuration of trait-based port scheduling is performed via a YAML config file, with each key correlating to a custom trait, and the value is a list of rules to follow to build a mapping. For instance, you could tell Ironic to attach_port when network.tag="green" && port.physical_network="green". Operators can also set a trait to use when none are provided. Here is an example of a trait-based networking configuration:

CUSTOM_DIRECT_ATTACH_A_PURPLE_TO_STORAGE:
  actions:
    - action: attach_port
      filter: port.vendor == 'purple' && network.name == 'storage'
CUSTOM_BOND_GREEN_STORAGE_TO_STORAGE_BY_2:
  actions:
    - action: group_and_attach_ports
      filter: >-
        port.vendor == 'green' && port.category == 'storage'
        && ( network.name =~ 'storage' || network.tags =~ 'storage' )
      max_count: 2
      min_count: 2

This port scheduling goes a step further with the ability to assemble dynamic portgroups. In Ironic, prior to Gazpacho, the only way to get portgroups – ports bonded together using LACP or similar technology – is to statically configure port groupings on a node. Now, with trait-based scheduling, you can assemble portgroups on the fly based on rulesets. For instance, not only could you attach green ports to green physical networks from the previous example, but by using the group_and_attach_ports action, you can have Ironic create a portgroup on the fly based on the specified rules – including the ability to fail processing if too few or too many ports would be included in the portgroup.

For full details, see the trait-based networking documentation.

Get started

Both the autodetect deploy interface and trait-based port scheduling are available now in Ironic 35.0 as part of the OpenStack 2026.1 (Gazpacho) release. Together, they represent a meaningful step toward giving end-users more control over how their bare metal instances are configured, without requiring admin intervention. Check out the full Ironic 35.0 release notes for everything that's new in this release.

Ironic's Networking Evolution: Bringing VXLAN to Bare Metal

Mon, 02 Mar 2026 10:00:00 GMT

The Gazpacho development cycle (OpenStack 2026.1) brings VXLAN overlay networking support to Ironic bare metal nodes. This enables bare metal servers to participate in the same L2 broadcast domains as virtual machines, providing better network scalability and improved experience.

Sometimes customer calls start with what seems like a vision, but turns out to be a misunderstanding of project documentation. It can be hard to tell at first, but this time once we understood the customer's perception, we quickly realized it was a misunderstanding with a vision.

In the past, a contributor from a networking hardware vendor posted a patch to enable Ironic ports to have configuration set which was to influence the OVN VTEP Controller and enable binding on remote devices. Once the project realized the scope and mode, we quickly backed out that change, yet as was once said, the die was cast. The idea of using Ironic nodes with VXLAN had been seeded into the universe.

And, it is not as if it is a bad idea. The biggest challenge large scale operators of sovereign clouds face is the management of the physical and the resulting logical networks and providing enough tools to address the bottlenecks as they are found. One of the biggest challenges with "classical" logical network management using VLANs is just the restricted number of logical networks (4096) which is due to the design of VLAN utilizing ethernet packet headers. This quickly grows into management of root switch bridges, spanning tree, and so on and so forth. Complex networks twenty years ago maybe had a few hundred VLANs, yet the proliferation of virtualization has driven the average enterprise's networking needs to the limits. Mix in scaled workloads, and traditional means of traffic management also no longer scale. This led to VXLAN being developed and it is the logical path forward for even mid-size cloud operators needing better network scalability.

For virtualization heavy environments, it made a lot of sense to just enable your workloads to have "virtual overlay" networks, and then map those virtual networks to the physical networks as needed. Soon hypervisors automatically established and managed "overlay" networking between hypervisors and virtual machines benefited by what becomes perceived as a limitless number of networks.

Where this gets "fun" and "interesting" is with Bare Metal and Ironic. My customer sought to put baremetal nodes into the same L2 broadcast domain of the virtual "overlay" network. Their perception was that because we had the one misleading feature, that we had generalized support. But with all things networking, the details can be nuanced.

Historically Ironic has had support for two basic networking approaches. The first is called flat, although it really is more "static" in nature because the cabling and port associations are all statically configured in an environment and the most advanced configuration case you may really leverage is routed spine-leaf handling of traffic. The second approach is the neutron network interface, which supports dynamic networking and the attachment of VLANs to bare metal nodes under Ironic's management.

Except, there is a limit to the number of VLANs on any given physical network. This VXLAN feature is not immune to the limitation even with its 16 million possible networks, but the odds of trying to operate 4096 networks on any single given physical device are also fairly minimal. A pragmatic approach was obviously required.

And so we got to work in the Ironic project and began to collaborate with cloud operators in the upstream project to come up with a solution to bring bare metal nodes to VXLAN networks. A few months later, we've merged a new mechanism driver called baremetal-l2vni into the networking-baremetal project, initial switch driver updates to enable VNI binding for logical switch devices into the networking-generic-switch project, and had even managed to get some initial testing in place.

What This Enables

This capability allows operators to place both virtual machines and bare metal instances on the same Geneve or VXLAN tenant networks. Previously, operators were limited to either setting a default tenant network type of "vlan" or manually creating and managing provider networks of type VLAN to facilitate bare metal node attachments. With VXLAN support, operators gain improved multi-tenant isolation at scale and dynamic network provisioning for bare metal nodes, removing the manual overhead of provider network management.

How It Works

In this model, Neutron Networker nodes (nodes which bridge networks and provide routing services through the bridging of the northbound and southbound database context of OVN) are able to attach Geneve and VXLAN networks to VLAN segments which are able to flow over a physical port and be translated across to the VXLAN network.

The switches leverage their ASICs to do the packet encapsulation and decapsulation work, enabling efficient traffic flows over the network. This hardware-accelerated translation means the performance impact is minimal while maintaining the flexibility of overlay networking.

The overall model enables switches to register the VXLAN VNI they require and allows traffic to be efficiently routed over the physical network. The same lower VLAN segment gets utilized on the remote switch which is then bound to only the VXLAN VNI and to the requested physical ports.

Effectively, VXLAN overlay becomes a giant cross-connect panel, allowing efficient and dynamic communication between a virtual machine and a bare metal node somewhere in a larger VXLAN enabled network fabric.

What's Next

While we're not "done", this next development cycle we'll shift gears to testing individual switches and their configuration. This may mean we identify some areas to improve or refine further, and hopefully will result in more verbose documentation as we go in case you decide you want to treat your DPU as a switch, or do some other sort of extended configuration. Our implementation expects VXLAN configurations to utilize BGP EVPN "ingress-replication" of type-2 routes, though we may extend support to multicast-style VXLAN configurations in the future. There is also an effort underway in Neutron to connect BGP Ethernet VPN tunneling to facilitate VXLAN connectivity. We expect our existing work to be compatible and largely just give operators the ability to choose the approach which makes the most sense for their environment, once that Neutron functionality is available.

Try It Out

This VXLAN functionality focuses on the switch-side networking attachments to VXLAN networks. Complementing this work, the Gazpacho cycle also brings Trait Based Networking decisions, which drive the early determination of VIF to Interface mapping - determining which interface should be used for network attachment. Together, these capabilities provide operators with greater flexibility in both interface selection and network attachments for advanced networking configurations.

This functionality is designed for fully integrated OpenStack deployments and requires Neutron. It is not currently applicable to standalone or Metal3 use cases.

Both networking-baremetal 8.0.0 and networking-generic-switch 9.0.0 are required for this functionality. The networking-generic-switch ML2 driver handles VNI management as ports are plugged and unplugged, ensuring the ML2 driver is aware of the flow, logic, and binding segment data necessary to facilitate network attachment.

If you're a vendor with a custom ML2 plugin and want to add support for VXLAN bare metal attachments, we'd be happy to collaborate on implementing the port attach and detach flows for your specific driver. Please reach out to the Ironic project!

We encourage operators to test this in their environments and provide feedback. Your individual configuration might be a little different or have a different model, and that is okay. We're up for patches and more collaboration as this is Open Source!

For more details, see:

Get Involved

Have questions or want to discuss VXLAN networking for bare metal? Join us in #openstack-ironic on irc.oftc.net or reach out on the openstack-discuss mailing list. We're always happy to help operators get started or collaborate on extending support to additional switch vendors and ML2 plugins!

Acknowledgments

Special thanks goes to Doug Goldstein with Rackspace for being an awesome community partner and helping push forward the VXLAN functionality.

Coming Soon - Threaded Ironic

Fri, 25 Jul 2025 20:00:00 GMT

To understand where you're going, you need to understand where you started.

We typically don't talk about eventlet. It was a foundational technology which OpenStack based on to address concurrency in a Python process. It enabled a level of concurrency which has simplified many things for OpenStack without developers needing to be mindful of threading. In essence, Eventlet replaces native threading and numerous blocking calls in a Python application. It also came with caveats, but it provided a suitable substrate to build upon.

For example, calling a time.sleep() call wouldn't actually block the application. The same is true for socket IO, and numerous other code paths which block a "thread".

This is because eventlet quite literally overwrote Threading logic with code which functionally turns the threads into coroutines, where any blocking action would result in different coroutines executing.

For OpenStack as a whole, this was fairly useful. Most activities were not bound by CPU resource constraints, even when a process was functionally single threaded for specific actions. Of course, some services decomposed furhter along logical boundries, but overall OpenStack services had found an balance in the process and task execution models which met requirements while also kept the design and maintenance relatively simple.

But time has finally come for OpenStack projects to remove eventlet. The reasons are complex, and truthfully eventlet has been holding OpenStack projects back, while changes in Python make the maintenance of eventlet problematic as time moves forward.

The end result, Ironic is becoming a true multi-threaded application. While not quite already released, expect Ironic 32.0 (part of the 2025.2 OpenStack release) to be the result of the hard work of contributors in the community.

Ironic and Threading

Ironic has had an interesting history.

A project which is widely used inside of OpenStack and as well outside of OpenStack.

A project which facilitates the day to day management of fleets of Bare Metal hardware.

A project which supports a variety of deployment strategies and performance profiles from embedded to heavily scaled out across an extensively large cluster.

The net result is you can run Ironic a single container, or in a distributed deployment, and the driving factor in how you'll make that decision is more an artifact of your operational requirements. For example, if you're using Redfish or IPMI, each of which have drastically different performance impacts.

With this, there are two activities within Ironic which are heavily concurrent and can be extremely resource intensive: Power state synchronization and Sensor Data Collection.

Both activities require the ability to interact with Ironic's database in a consistent way, where any locking is able to be updated inside of the database utilizing the existing connections as to also not risk a possible split-head situation through database cluster. Combined with the consistent hash ring which assigns nodes to "ironic-conductor" processes, this model avoids possible deadlock conditions for a node, and when combined with the pre-existing locking model, work on a specific node is reserved.

But going back to the two tasks which generate the most activity for an Ironic deployment, the power state and sensor data processes, each process ultimately can become IO bound. They can, depending on the driver, launch external processes, or have a substantial amount of structured data to parse in order to achieve the desired end result.

In the past, Ironic has handled this concurrency with eventlet. Some work would be temporarily blocked for a short time, but the threading model allowed for the work to be managed through a pool of threads where specific work is assigned to a thread based upon the user requests or actions, the vast majority of it being asynchronous. Moving forward, native python threads will be used to facilitate the removal of eventlet.

This unfortunately changes the process model, to an extent, as well as the amount of memory, and ultimately the overall behavior pattern of Ironic. While there are many changes, the results are also positive. In large part because threads align with the process design model in use. Work becomes a task on the thread pool, and the end result is the database record is updated. A substantial portion of already existing code helps provide the guard rails to prevent issues like zombie or orphaned threads, supported by the overall design of Ironic. Where things are going to be most interesting, is where tasks previously hung before, in the same way, for example when a remote BMC is unreachable.

To take a step back though, threads are often viewed with skepticism and confusion. In part because because it is hard to mentally think of the behavior patterns which result. On a plus side, risk of the overall change is greatly reduced but not eliminated due to the removal of eventlet. Even amongst Ironic maintainers, we had to take a step back and look at other common everyday applications and their thread behavior to help reset and ground our perceptions.

Operator Impact - Process model changes

So far, we've already discussed, at a high level, the fact that the removal of eventlet will result in the use of Ironic becoming a multi-threaded application. A side-effect of the changes, is that each "application" being launched becomes a sub-process from the launching process.

The easiest way to think of this is as a "launcher" and the "service". Some common code exists between the two, for example shutdown handling must be triggered by a launcher because threads cannot receive signals. Aside from some additional complexity around stopping/starting the processes, the net effect is an increased memory footprint.

Ironic also has multiple use modes, depending on the footprint of the deploy. For Example, the Metal3 project uses Ironic under the hood in a single container which operates a single process. Moving forward, this single container will now have three processes. The first process, the launcher will be what the container starts, and then two sub-processes will spawn. One sub-process will be the API surface for Ironic, and the other sub-process will be the conductor process which is charged with doing the heavy lifting of the deployment. This also means that some drivers or processes may then fork additional processes off from a thread to achieve additional actions like speaking to a device using ipmitool, or when converting an image using qemu-img.

The traditional API service will still be available for Ironic, as well as the conductor process, but they will be threaded. The Ironic project decided that because our overall model was overall not impacted by these changes, that it made the most sense to not try to maintain some level of backwards compatibility where eventlet could continue to be used. The new model, from our early benchmarks does appear to be similar or even more performant, although we are under no allusions that we may find issues moving forward related to these changes, but those issues will need to be worked as they are discovered.

At the end of the day, we see this change keeping a similar operational profile and experience for the user. It is just they will notice it if they look at the process list.

Operator Impact - Memory Usage

But when looking at a process list. Operators are very likely to notice an increased memory footprint.

There is not much we can really do about this, and we're sorry. But we'll try to explain.

In Linux, memory reported for an application is in one of two buckets: Virtual Size (VSZ), and Resident Set Size (RSS).

The KEY difference between the two is that the Virtual size represents what the application has allocated or mapped, but not necessarily used. The Resident Set Size is what is actually used for the application to operate.

Threaded applications often require more memory. Part of it is the thread stack for thread local memory needs. At the same time, open files or memory ranges across threads cause the the VSZ numbers for processes to increase in line with the size of the thread stack memory allocation size and the workload of the application. This is linear in nature, but does closely track with the threading and we identified this in our early benchmarks where we were not getting rid of old threads. We simply kept adding new threads until the overall application started to have performance problems. We worked with some other community developers to propose a new thread pool model which kept our thread counts in line, and ultimately our memory utilization under control. The downside of the entire model with a "launcher" and "service" process, is that each instance of such has a memory footprint and ends up with distinct independent copies.

The net result -- 10x increase in VSZ counts. 2-3x increase in the RSS counts. This will require operators to tune any memory constraints where they run Ironic in a Container or where they may be applying strict limits to the process runtime. At the same time, the process concurrency improvements are going to also result in a faster Ironic deployment, overall.

Will I need to tune this newly threaded world?

As a project maintainer, we started this journey with an expectation that we would likely need to tune the threading model. Limit our number of threads. Add more capabilities to stop and resume work asynchronously! Who in their right mind would have 300 threads anyway! Turns out, we would! And it also turns out many applications already use a huge number of threads. Compare to a Slack client or Web Browser. Modern machines have thousands of running threads today which largely goes un-noticed.

So the short answer is you may not need to tune Ironic, unless you have extremely high concurrency and lots of slow responding BMCs. Another possibility is that your workload is intense and a conductor is overloaded. All of these scenarios are very situational, so we'll try our best to set some context.

A stock Ironic deployment in this should have approximately 30-40 threads running by default. This should account for power state synchronization and the various periodic tasks which trigger every on a regular basis. Ironic has, as an artifact of design, a thundering herd problem which can occur. Arne Wiebalck with CERN talks about this to an extent on youtube, but this tends to smooth out as time progresses.

This can compound if you turn on sensor data collection. And be made even worse is you have hosts which take a long period of time to respond. Ultimately, this does boil down to needing to know your situation. Moving forward, Ironic is going to also begin to sanity check some of the runtimes of work, and try to get into a habit of making suggestions in the logs as warnings for operators to evaluate tunable settings. Until then, we'll cover some of the key tunable options to be aware of.

Setting [conductor]setting workers_pool_size

The Conductor configuration option, workers_pool_size is the total number of conductor workers which is divided into a reserved pool, for mission critical actions, and a normal workers pool. This directly impacts the maximum number of threads, but generally the hope is this value is good for most deployments and this should be only modified if the thread pool is running out of threads.

Setting [conductor]periodic_max_workers

The conductor has an additional setting, periodic_max_workers, where it limits the number of concurrent works for periodic tasks. By default this is limited to 8 workers, and will also automatically reduce the power state workers if also modified. If you have a large fleet and you need to increase your parallelism for periodic actions, this is a good starting point.

Setting [conductor]sync_power_state_workers

The number of threads which are launched for synchronizing the power state is controlled by the sync_power_state_workers option. A constraint is that the periodic_max_workers setting must be at least the same value, otherwise the workers may be restricted. If you have slow responding Baseboard Management Controllers, you may want to increase these settings.

Setting [conductor]sync_power_state_interval

The interval, in seconds, between launching new attempts to check and synchronize the power state of remote hosts with the database. This option, by default, is set to every sixty seconds. While multiple sweeps cannot run at the same time, the default setting may be a bit aggressive for larger clusters.

Setting [sensor_data]workers

This is the maximum concurrency for sensor data collection by a conductor. Sensor data collection, depending on the management protocol or even the performance of the remote BMC, can wildly impact the overall performance of this process. For example, a single host may take 30-60 seconds to query overall. Obviously, if your collecting sensor data, you may want to increase the worker count, and appropriately set the frequency in which you collect sensor data.

Timeout and Retry Settings

A trend we have seen over the years is some operators wish to avoid all possible failures by extending timeouts. This is extremely counter-intuitive because timeouts should generally be shorter, as each thread stuck waiting to timeout is a consumed thread. If your trying to tune, do take a look at your timeout settings and consider 30 seconds instead of 60 seconds. Consider less retries in general.

The Future Beyond Threading

As a project, we recognize we have two distinct challenges which threading helps, but also doesn't help. Blocking IO, and the thundering herd of periodic tasks.

Moving forward, we expect to take a more hybrid approach where we are using coroutines in threads to increase the parallelism upon which we perform certain actions like power state sync or sensor data collection.

For periodic tasks, we're likely to move to a model where we keep a list of delayed future actions which need to occur, and utilize a single periodic task to spawn these actions as follow-ups. That said, both of these changes are substantial changes beyond the removal of eventlet. In a weird sense, they are actually more complex changes to Ironic's architecture than just removing eventlet and getting threading as a side effect.

Thanks

A special thanks goes to all of the Ironic contributors who have worked some long hours over this past development cycle to help ensure we can remove eventlet. We're almost there, and hopefully it will be gone in the next few weeks!

Also, special thanks goes to the current Eventlet maintainers and their active efforts to wind down Eventlet. You can learn more at their Eventlet Removal website.

Bare Metal SIG meetup 2023Q1

Wed, 21 Dec 2022 10:55:20 GMT

New year - new Bare Metal SIG meetings! Starting with 2023, we're switching to a quarterly cadence, while making the meetings longer, more entertaining and opening the floor for more active engagement. The first event will take place on Wednesday, Feb 8th, from 15:00 to 17:00 UTC. We will use Zoom, the link can be found in the mailing list post.

The agenda looks like this:

Meet'n'greet: news from Bare Metal SIG, Ironic and the ecosystem. [15 minutes]
Panel discussion: things you didn't know Ironic can do for you. Are you doing something absolutely crazy with Ironic? Have you ever been afraid to get kicked from IRC for even asking about the use case you have? Time to shine! Come and show us that Ironic is not only about provisioning qcow2 images through PXE. [30 minutes]
Panel discussion: scaling bare metal provisioning (Ironic and beyond). How do you make these racks of servers buzz together? Come talk to experts from CERN, G-Research Open Source Software team, and Red Hat. Bonus from us: introduction into the new "sharding" feature! [30 minutes]
Open floor. Any topics welcome, but the suggested topic of the day: battle stories! Share your most crazy bare metal craziness for good laugh and education.

If you have any questions, please reach out to Arne Wiebalck (arne_wiebalck), Dmitry Tantsur (dtantsur) or Jay Faulkner (JayF) on IRC.

What is after BIOS?

Mon, 04 Apr 2022 20:00:00 GMT

Every six months or so, the Ironic project community meets to discuss current and new topics of interest. We call this the Project Teams Gathering. This time serves as a time for many projects to gather and have the important discussions, some of which may overlap into other projects, or may impact or affect numerous projects. Regardless if a project treats the Projects Teams Gathering as a formal event, or a set of hallway track style discussions, often we keep notes, and we try to bring others into the discussion. It is part of our community's culture.

And today, an interesting topic came up when we were performing our retrospective:)What is (life) after BIOS?

When we say BIOS, we can mean many different things. In this case, we're referring to BIOS booting. Most often referred to as "Legacy BIOS" booting.

Legacy BIOS booting is the way computers, specifically IBM PC and compatible systems started at power-on which proceeded from basic device initialization, firmware loading, and then initiation of the operating system boot using a Master Boot record on a storage device, which then redirects to a boot loader, which then may load an Operating System kernel, such as Linux, which is then able to launch the rest of the operating system.

Much of this works through special locations identified and expected on disk, but not through an understanding of the contents. Which means if you accidently deleted a critical file, and then restored it without rebooting, the likelihood your computer would no longer boot was quite high. Given much of this technology was rooted in fundamentals started in the 1980s, much room for improvement followed.

But for those of you that don't remember those days, think of how to start an operating system with 1 megabyte of address space.

##UEFI is now##

The Unified Extensible Firmware Interface, better known as UEFI, was the answer. While initial work in replacing Legacy BIOS booting started in the late 1990s, it did not pick up steam and adoption until 2010-2020. This spread not only to technology based on the Intel x86 architecture, but also similarly in the ARM architectures as well, and is seeing further extension to additional architecture because there are far more advantages to this mode of booting. No more 1MB memory limitation, and now in theory you could boot Linux inside of your EFI pre-boot environment before you boot your final Linux kernel.

Frightening right?

Well, don't worry! It is actually possible to boot a system from an EFI shell, or explore the contents on disk to discover what exactly is going on. And if your lucky, you may have some graphical tools available as well before your computer is even running an operating system. Plus! Aspects like Secure boot are further baked into the system, as opposed to the predecessor Trusted Boot.

##Why does this matter with Ironic?##

One of the things the Ironic community is sensitive to is changes in the Server hardware marketplace. Ironic is used by hardware and software vendors to deploy their solutions for their clients much as a cloud operator deploys workloads. So when a vendor like Intel announces they are going to stop supporting Legacy BIOS boot, we take notice. Except it didn't happen quite as rapidly as Intel proposed.

During the Wallaby development cycle (1st half, 2021), some Ironic cores finally started to see cases where users were interacting with brand new Data Center server platforms where Legacy BIOS boot was no longer a feature. Where you couldn't boot to it, but maybe you could still invoke the 20h interrupt to attempt to load Option Roms from hardware... where that might freeze the system. It was an interesting few months to say the least, yet the consensus was reached that "Time to default Ironic to UEFI" had come.

And so in the Yoga release of OpenStack Ironic, the default was switched.

And now the quest is onward!

##So what is next?##

While I'm sure many would prefer to have crystal balls and be able to make sense of the long term plans of the community, we just don't have such magical power. What we do have is foundations of technology upon which we can build the next evolution.

And emerging work by Trammell Hudson may be a path we should explore. An in-UEFI Ironic Agent able to deploy and then boot to an operating system without a reboot would be an amazing feat, and I personally think it would be an excellent project goal. Think of all of the possibilities! While, we might not be able to entirely rely upon it for aspects quickly like cleaning or full introspection, we may be able to one day.

And then there are additional CPU architectures, but it would mean a lot for the community if operators interested would be willing to help enable hardware access. Even if the possibility or hope is raised to the community, it becomes context to drive us. A mission to take on. A barrier to scale!

And there is the old conundrum of the computer, inside of the computer. We once viewed this as devices with firmware, but now entire operatings systems can live on a card and do distinct work. Hopefully, forward progress will be ma de here so we can do the needful, and enable the automation of even more infrastructure.

The scale of usage

Wed, 05 Jan 2022 22:00:00 GMT

The Scale of (Ironic) Usage

About once a year, often in the beginning of the year, I receive a question to which I cannot really answer. Truthfully, it often comes from several avenues, and different organizations.

It is not that I don't know, we as a community do know... kind of. But there is a conundrum.

Open Source contributors representing firms cannot speak to many of the details because revealing the context behind discussions and associating them with individual's or organization's name is inappropriate unless they had explicitly granted permission. And even then, they are the exception to the rule which is to preserve the privacy of our customers and users. Those deeply engaged in a community build a relationship of trust, and some additional context gets conveyed. Lingo, or Jargon as it was once called, gets used. We paraphrase, re-scope, and obscure much when making our arguments for or against a given aspect. But we're all trying to do the right thing, for revealing the details is really, truly, in-appropriate.

Which is truly also kind of remisicent of a recent GCC bug filing.

Which sure seems like a similar conundrum, and establishing the context of who the customer happens to be.

Artifact of infrastructure

The closer you get to lower layers of intrastructure, the more important it becomes. Also, the greater impact one can have should details make it into the eyes of those who would do harm.

Typically in communities, we can see the power lines, we might be able to figure out where the water lines are precisely, but the closer we get to those infrastructure centers, the more secretive or restrictive those environments need to be to ensure the service and security of the services being provided.

It is the same for cloud infrastructure operators, telecommunications providers, and even train operators. They provide a substrate service to meet the needs of their customers. Often, the customer never knows how the server got there. Or how their telephone call completed, or what drove the spike into the railroad track tie.

On some level, that is an implementation detail on a path.

It is hard for vendors as well

It is also not the easiest thing for vendors who specialize in providing support or development services to even have these answers either. As an entity, they are bound by their agreements and consideration of the customer's privacy. Often reports and telemetry data, due to information security concerns, come from reproduction or internal test environments, and not actual production environments where an issue is being encountered which requires assistance or support. Ultimately, sometimes the hardest thing to wrap one's head around in these sorts of situations is that there is often not just a singular customer environment, but many.

Vendors are in a very similar place to the contributors. They serve as proxies for their customers at the end of the day.

The end result

The result of this is a lack of a clear picture of usage. It is difficult to truly understand the users of any given piece of Open Source software.

But, we can piece together public context, and this is why the OpenStack Baremetal Logo program is helpful to the Ironic community. It helps highlight the firms which have an interest in the software. To join the program, each organization was required to submit their use case in how they use ironic. A number of them resulted in blog posts and stories. While this doesn't always result in explicit counts of number of servers, it is only the portion who actively stepped forward.

Another avenue for gauging interest and usage of an upstream project is through who contributes to the project itself. We don't necessarily have context into their usage or use case unless they speak of it publicly, but a commit from a firm may be be driven by any number of different drivers: Customer needs, Internal Need, Bugfixes from internal or external usage, Compelled need to fix a typo, or a friend just wanting to make sure they can have breakfast on occasion with another friend resulting in a pain point being fixed.

The following screen shots are from Stackalytics conveying commits by companies over the current Yoga development cycle and the two prior development cycles.

Unfortunately, without diving into each and every single change and trying to determine the motivation behind each, it is difficult to discern the interest and driver.

And as humans, we tend to put more weight behind direct first person statements, especially when we begin to talk about a scale of use or a size of a deployment as direct statements from humans also generally have a scope in which they are spoken which helps convey the full context. This could be the question which prompted the response.

I'm an infrastructure operator, How can we help?

There are multiple things you can do to help spread the word and context of usage.

The first is to participate in the OpenStack User Survey. This helps provide the Open Infrastructure Foundation information which helps put into focus the state of the ecosystem and it's usage.

The second way you can help is to engage the community directly and share your story. We have a Bare Metal Special Interest Group you can join and we will be happy to discuss topics, provide guidance, and this is the most effective path for feedback and context sharing directly to the developers.

The third, and path only for the most brave is to submit a blog post. If you can't share your story directly on our blog, we are always happy to share a link.

Ironic 18.2

Fri, 24 Sep 2021 16:00:00 GMT

Xena Release

On Wednesday, September 22nd, 2021, the Ironic team released its OpenStack Project "Xena" cycle deliverable for Ironic as version 18.2.0.

For those who are not aware of how OpenStack branches are supported, this will be a stable branch which will be maintained for a period of time, as opposed to an Intermediate release by the Ironic project which may only see backports for a limited amount of time.

Overall the Xena development cycle consisted of over 22 new features to Ironic, and over 48 thousand lines of code having been modified.

Notable Features, Fixes, and Changes

This list includes Ironic 18.2, 18.1, and 18.0.

Parallel image downloads are now enabled by default.
Event subscriptions can now be created on Redfish BMC's through the vendor pass-thru interface.
boot_mode and secure_boot are now top level fields of a node which can enable easier management of these states. Some additional work into this should be expected as time goes forward.
Node History events are now recorded, and can be retrieved by the API. Expect CLI tools to be updated to have functionality for this early in the OpenStack Yoga development cycle.
PXE loaders can now be configured to copy files from the base operating system. You must specify this, as Ironic does ship defaults at this time. Look for [pxe]loader_file_paths in the configuration.
The anaconda deploy_interface can now post configuration drives.
Vendor specific driver_info parameters for items such as deploy_kernel, deploy_ramdisk, deploy_iso, and rescue_iso have been deprecated in favor of vendor-less parameter names.
Performance issues when retrieving lists of nodes have largely been addressed. This was written about in prior blog posts, and should greatly help Nova integrated deployments.
Bios setting registry fields are now available when retrieving the BIOS settings.
A new custom-agent deploy interface can be used for instances where the agent provides all necessary deployment steps.
Deprecation warnings for Secure RBAC related policy changes are now suppressed. This was largely because the larger OpenStack community was not able to implement delineated System and Project scoped rule sets before the end of the Wallaby development cycle, nor the now completed Xena cycle. At present all projects are not expected to be in a position to deprecate legacy rules until the end of the Yoga development cycle. Stay tuned!

Please remember to update your agent, or you will be missing the following features:

bmc_mac is now stored in Introspection data.
Disk, Network, CPU, and Memory burn in cleaning steps are now available for optional stress testing of these components. Special thanks goes to CERN for contributing these to the community.
Bootloader CSV files for UEFI defaults can now be loaded from EFI partitions and used to set the appropriate bootloader. While this capability was backported to address operational issues being encountered by users, this is an important step as Ironic moves forward into the world of being UEFI native and UEFI being the default.

And more! Please review the release notes for even more detailed information.

Where can I get it?

Ironic 18.2.0 can be downloaded from a variety of places:

PyPi
OpenDev
"git clone --branch stable/xena https://opendev.org/openstack/ironic.git"
RDO Project xena/stable repository (Untested)

This blog post was written a few weeks prior to the official Xena release of OpenStack, so expect distribution packagers for OpenStack to pickup this version of Ironic as time moves forward.

Looking Forward

Soon into the next (yoga) development cycle, the Ironic project expects to switch the default boot mode for all deployments to UEFI, if not otherwise already specified. We anticipate this default change to take place with future version 19.0.

Such a change can potentially be breaking for many deployments if they never specified a default previously, but it is far past-time for ironic to move away from a default of Legacy BIOS booting by default. We are not expecting to remove the support anytime soon, so operators will just be able to assert the bios default instead if they, or their hardware is not ready for this change.

Questions? Want to help?

Feel free to join us in #openstack-ironic on irc.oftc.net if you have any questions!

Ironic also has its Project Teams Gathering upcoming during the week of October 18th, 2021.

Performance Update

Thu, 05 Aug 2021 22:00:00 GMT

What a Journey

When I started out on a journey of trying to improve the performance of ironic a few months ago, the journey ended up taking a bit of a different path from what I expected. But I can now say with confidence, because all of the patches have merged and even backported to our "Wallaby" release, that we've made Ironic faster.

The path was similar to what was expected, but half of the effort became building consensus and sharing context. Plus, very verbose notes in code do tend to help. Along this path, we also learned some new aspects, and were able to incorporate changes to hopefully improve the overall situation for most operators.

Where can I get this?!

Expect this in Ironic 18.1 or Ironic 19 (Xena) (whichever ends up being the next release).
Alternatively, Ironic 17.0.4 (Wallaby) since we backported most of the changes, and you can manually add database indexes if you need them.

If you need some of these patches backported to older releases, join on us irc.oftc.net in #openstack-ironic and let us know.

Where did we end up?

Overall, we have been able to reduce the time it took to download our sample data by 82% and improve the number of nodes returned per second to consumers like Nova who request lists of nodes with distinct fields in excess of 630% from our very first performance test measurements of Ironic while in the process of exploring performance bottlenecks and how to address them.

This whole effort challenged how we approached operations with node lists.

Additional database indexes
Improvements to how we sanitize nodes
Improvements to how we handle the access control context when lists of nodes are being handled.
Removal of excess work from the Database, and Database client

These changes show an increadible performance improvement over Ironic 17.0 which was released before we started this effort to improve performance.

One size, does not fit everyone

Ironic fulfills the needs of several different use cases. From a Systems Administrator just needing to deploy the same OS image to a number of machines in fairly rapid order. To software deploying specialized ramdisk images and configuration such as those that may be used to stand up a small private cloud. To the largest and most complex deployments where Ironic powers the management of all Bare Metal nodes and resources within an organization.

Not only does the scale differ, but so do the usage patterns, and ultimately the resources required to continue to manage a dynamic Bare Metal cloud. In this exercise, this was something we came to learn with our code.

We treated database queries as one size fits all, and to put it blunt, they cannot be treated as such. Without writing more on the subject than anyone would want to read, there are trade-offs. Ironic now attempts to leverage the trade-off when lists of nodes are being requested which should improve general node list performance. The trade off being in those cases, Ironic will no longer use a database join to pull in a composite field for a node. This is because that join was actually triggering additional overhead in the SQL client as the result set was being downloaded and converted for use by the program. Basic performance testing shows this change to be on-par with when we utilzied joins, but it should help operators who use one or more node traits assigned to a node.

Next Step

Now that we have have some awesome performance improvement merged to Ironic, there are obviously several future steps we can take. Only time will tell if we will take them.

Make chassis_uuid use a join - Presently, explicitly asking for a node's chassis_uuid is kind of slow. Unfortunately this is an object model change and cannot be backported.
Make allocation_uuid use a join - Explicitly asking for an allocation, also causes some additional overhead query. Unfortunately, this is also an object model change and cannot be backported.
Improve Nova's interaction with nova.virt.ironic - This is not explicitly in Ironic, but the driver for Ironic in nova is subjected to some calls internally that make sense when your nova-compute service is a hypervisor, and not a transport through to Ironic. Specifically bug 1923955 is one of those cases. We believe this can use the internal cache.
We may explore our periodic task handling, but at present this does not feel like it would have much of an operational gain for users of Ironic.

If your hoping to see a performance improvement when retreiving the managing conductor for each node via the API, I wouldn't expect that unless we consider additional data model changes. But if you have a use case, please bring it to our attention and maybe it will inspire us!

Feel free to join us in #openstack-ironic on irc.oftc.net if you have any questions!

Bare Metal SIG - Introduction to Bifrost

Mon, 19 Jul 2021 10:00:00 GMT

Bifrost allows to install Ironic in standalone mode (without other OpenStack components) and is also often chosen as a starting point to get familiar Ironic, or even for Ironic development. Core contributor Julia Kreger gives us an introduction to the Bifrost project, its history and how to use it.

The search for performance

Wed, 05 May 2021 22:00:00 GMT

All the context

I recently began on a journey to discover performance issues in Ironic. In part, because I made changes to improve operational security capabilites to users which we knew would have some impact and our amazing friends at CERN encountered an unrelated performance issue.

This somewhat sent me on a mission. A mission to improve performance!

But everyone's performance issues can vary based upon multiple factors.

It is one of the reasons we don't talk about scaling. We will talk about the rich concepts and capabilities available to scale a deployment, and intend to write about this soon, but what people tend to seek is aspects like raw counts of nodes per conductors. But does the conductor have multiple CPUs or one CPU? Are you using IPMI or Redfish? Is your storage performant... or not? Are you using a centralized message bus, or is everything just installed on one laptop that is toated from rack to rack. All of these things drasticaly impact performance in different ways.

Unfortunately, Ironic has to take a solid portion of blame for performance. Maybe blame is the wrong word. The application architecture supports the ability split and distribute distinct tasks while also offering the capability to support rolling updates and online data migration between versions. At the same time, the project and it's capabilities have evolved organically over time.

And, in that organic evolution, we never built tooling to provide developer and troubleshooter friendly tools to make it easier to evaluate some of these things and comprehend the enviroment and it's data. And I wish we had these tools, of course, in the end, I had to add numerous print statements into the code to precisely time exactly what is occuring. But I'm alluding too quickly.

Disclaimer

This exploration took place on a version of ironic from the upstream development branch released after the release of Ironic 17.0. I anticipate fixes noted in this to be in Ironic 18.0 which we're hoping to be released soon, and backported in the future.

The journey to improvement

In starting, we knew of an obvious performance deficit. The database, query performance, and resource contention between conductors. This somewhat was led by the issue raised by our friends at CERN where the internal task model was causing additional queries to be exectued automatically for objects which were not used. This was re-tooled to lower the average concurrent query count amonst their twenty-plus conductors managing their baremetal hardware.

I'll present a mild warning at this moment, all numbers below are from a virtual machine on my personal desktop. I may, or may not had a StarShip launch stream running in the background on this computer, but I was able to reproduce these numbers under a few different conditions, even with the headache of Thermal Throttling was a thing.

Database Indexes, Result sets, and SQLAlchemy

And so, I started looking at adding indexes. This was surprisingly easy for me as someone who has spent substantial time working on databases. The headache was creating some mock data to exercise this, and then measuring it.

Then I had to remember the way SQLAlchemy worked when you use an Object Relationship Model. Long story short, it downloads the entire object, and then assembles the object. So indexes only help so much. They help for query concurrency and raw result set generation. And then I remembered, exactly why they say to never run "select *" on your SQL queries, because all of the columns are collected for the result set.

The answer was to restrict the query down to specific columns. Turns out, that is not entirely honored in Ironic's database access code. This shall be fixed.

Red herrings of Objects

When consulting the SQLAlchemy documentation, it is easy to to jump down the path of thinking that all of the object generations are part of the latency between getting the result set, downloading the result set, and then converting the result set to objects, which are then transformed and supplied to the API consumer.

Exploring this, and after numerous print statements into the code, it became clear the policy enforcement and sanitization code uses a substantial portion of the time to generate and return the result sets.

And then I started beating myself up, since this was code I recently modified. I created part of this issue, or at least compounded it further.

So I have a path forward, to just keep things simple and only do the needful, not extra work. It turned out taking this approach makes things even faster. We tend to error on the side of caution. To always expect to do certian tasks, but rarely stop to check if we need to take an extra step.

End results

In the end, It appears we're able to squeeze out quite a bit more performance out of the API through a series of changes and improvements. Some of these these changes may not be feasible. Some may take a little more work. Most should be able to be backported to older versions, but that remains to be seen. In the mean time, I'll leave you with a graph.... including a little more fine tuning and addressing the afternoon heat causing some thermal throttling.

And now, you've read the story of how I found a way to improve Ironic's performance dramatically!

Feel free to join us in #openstack-ironic on irc.oftc.net if you have any questions.

Checkout Case for Ironic in Cern IT

Fri, 30 Apr 2021 10:00:00 GMT

Ironic contributor Arne Wiebalck writes in a recent blog update on the CERN TechBlog about why they use Ironic and how it helps them support their users and ultimately their mission.

Bare Metal SIG - Secure RBAC

Tue, 20 Apr 2021 10:00:00 GMT

Ironic contributor Julia Kreger gives us an overview of Secure RBAC, a Wallaby cycle community effort, in Ironic.

Bare Metal SIG - Ironic Prometheus Exporter

Tue, 20 Apr 2021 09:00:00 GMT

Ironic contributor Iury Gregory Melo Ferreira introduces us to the Ironic Prometheus Exporter, a utility to help expose Bare Metal node sensor data to Prometheus.

Checkout Injecting Files in Ironic

Wed, 24 Feb 2021 15:56:14 GMT

Ironic contributor Dmitry Tantsur writes in a recent blog update on his blog how to use the new injecting files feature in Ironic.

Bare Metal SIG - Deploy Steps Introduction

Wed, 10 Feb 2021 09:00:00 GMT

Ironic contributor Dmitry Tantsur introduces us to the Deploy Steps, a mechanism to have customized deployments processes.

Checkout Deploy Steps Tutorial

Tue, 09 Feb 2021 13:02:47 GMT

Ironic contributor Dmitry Tantsur writes in a recent blog update on his blog how to create and use in-band deploy steps with Ironic.

Bare Metal SIG - Redfish Interop profiles

Tue, 02 Feb 2021 11:00:00 GMT

Ironic contributor Richard Pioso shares about the effort to create Inter Operability profiles for Redfish.

Bare Metal SIG - Ironic/Neutron ML2 interaction overview

Tue, 02 Feb 2021 10:00:00 GMT

Ironic contributor Julia Kreger shares an overview of the interaction between Ironic and Neutron with ML2 plugins.

Bare Metal SIG - Multi-Tenant Ironic

Tue, 02 Feb 2021 09:00:00 GMT

Ironic contributor Tzu-Mainn Chen shares an overview of the mutli-tenancy model supported in Ironic.

Checkout Ephemeral Workloads with Ironic

Tue, 19 Jan 2021 14:56:12 GMT

Ironic contributor Dmitry Tantsur writes in a recent blog update on his blog how to use the ramdisk deploy interface to run ephemeral workloads with Ironic.

SuperUser - Scaling Bare Metal Provisioning with Nova and Ironic

Mon, 11 Jan 2021 10:00:00 GMT

Arne Wiebalck and Belmiro Moreira with CERN and Sunny Cai with the Open Infrastucture Foundation recently posted a an article on SuperUser titled Scaling Bare Metal Provisioning with Nova and Ironic at CERN: Challenges & Solutions. In this post, they talk about their experiences scaling their deployment and the solutions which they found to the problems they encountered. It is a very informative piece, check it out!

Checkout Ironic 2020

Fri, 01 Jan 2021 10:00:00 GMT

Ironic contributor Dmitry Tantsur has taken some time to share with us his recollection of the most important events from 2020 in the Ironic world. Check it out!

StackHPC shares about Software RAID

Mon, 18 May 2020 10:00:00 GMT

In a blog post we recently found, the awesome folks at StackHPC have written about Software RAID with Ironic. Special thanks goes to Stig Telfer and Doug Szumski!