RackN founding member of LF Edge

Today, the Linux Foundation announced LF Edge to help build open source platforms and community around Edge Infrastructure. We’re just at the start of the effort with projects and vendors working in parallel without coordination. We think this LF Edge initiative is important because the Edge is already a patchwork of different data centers, infrastructure and platforms.

RackN has been designing our DEEP Edge federated control around open Digital Rebar because we know that having a shared abstraction at the physical layer makes everything we build above easier to sustain and more resilient.

We’re excited to be part of this effort and help that you’ll collaborate us to build some amazing Edge capabilities.

Note: We’re also watching the OpenStack Foundation work with Edge also. Hopefully, we’ll see collaboration between these groups since there are already overlapping projects.

0 comments on “DRP v3.11 PROVISIONS WITHOUT REBOOTING”

DRP v3.11 PROVISIONS WITHOUT REBOOTING

Some features are worth SHOUTING about, so it’s with great pride that I get to announce DRP v3.11.

The latest Digital Rebar release (v3.11) does the impossible: PROVISION WITHOUT REBOOTING.  Combined with image-based deploy and our unique multi-boot workflows, this capability makes server operations 10x faster than traditional net install processes.

But it’s not enough to have a tiny golang utility that can drive any hardware and install any operating system (we added MacOS netboot to this release).   RackN has been adding enterprise integrations to core platforms like Ansible Tower, Terraform, Active Directory, Remedy, Run Book and Logstash.

Oh!  And checkout our open zero-touch, HA Kubernetes installer (KRIB) based on kubeadm.  We just added advanced Helm features for automatic Istio and Rook Ceph examples.

To see more: https://github.com/digitalrebar/provision/releases/tag/v3.11.0

0 comments on “May 5 – Weekly Recap of All Things Site Reliability Engineering (SRE)”

May 5 – Weekly Recap of All Things Site Reliability Engineering (SRE)

Welcome to the weekly post of the RackN blog recap of all things SRE. If you have any ideas for this recap or would like to include content please contact us at info@rackn.com or tweet Rob (@zehicle) or RackN (@rackngo)

SRE Items of the Week

RackN Announcement
[PRESS RELEASE] RackN Ends DevOps Gridlock in Data Center  

Today we announced the availability of Digital Rebar Provision, the industry’s first cloud-native physical provisioning utility.  We’ve had this in the Digital Rebar community for a few weeks before offering support and response has been great! READ MORE
_
______

Cloud Native PHYSICAL PROVISIONING? Come on! Really?!
 By Rob Hirschfeld

Today, RackN announce very low entry level support for Digital Rebar Provisioning – the RESTful Cobbler PXE/DHCP replacement.  Having a company actually standing behind this core data center function with support is a big deal; however…

We’re making two BIG claims with Provision: breaking DevOps bottlenecks and cloud native physical provisioning.  We think both points are critical to SRE and Ops success because our current approaches are not keeping pace with developer productivity and hardware complexity. READ MORE

RackN @ DevOpsDays Austin

IMG_0810

Slides from Rob Hirschfeld’s talk – The Server Cage Match

SRE vs DevOps vs Cloud Native: The Server Cage Match by Rob Hirschfeld

I don’t believe in DevOps shaming. Our community seems compelled to correct use of DevOps as an adjective for tools, teams and teapots. The frustration is reasonable: DevOps clearly taps into head space for both devs and operators who see a brighter automated future together. For example, check out this excellent DevOps discourse by Cindy Sridharan.

As an industry, we crave artificial conflict so it’s natural to try and distill site reliability engineering (SRE), DevOps and cloud native into warring factions when they are not. They all share a focus on Lean process. READ MORE

SRE News

What is DevOps? By Cindy Sridharan @copyconstruct  
https://medium.com/@cindysridharan/what-is-devops-5b0181fdb953

It happened again this week.

At this Wednesday’s Prometheus meetup I was hosting, I asked one of the attendees what he did for work.

He looked at me briefly before he barked one word in reply — DevOps — and then promptly made a beeline for the pizza at the back of the room. READ MORE
________

An Influx of Kubernetes Installers Raises Questions Around Conformance
https://thenewstack.io/kubernetes-installer-explosion-natural-enthusiasm/

For the Kubecon Europe last month, industry observer Joseph Jacks pulled together a list of over SIXTY (yes, 60) Kubernetes installers and services. This wealth of variation that made itself known as the conference, happily, kicked off a conformance effort to ensure that users get a consistent experience. I’m a strong believer that clear conformance builds ecosystems and have deep experience working on that from my OpenStack DefCore efforts.

In short, conformance is not a vendor issue: it’s a user experience and ecosystem issue.  READ MORE

UPCOMING EVENTS

Rob Hirschfeld and Greg Althaus are preparing for a series of upcoming events where they are speaking or just attending. If you are interested in meeting with them at these events please email info@rackn.com.

OpenStack Summit : May 8 – 11, 2017 in Boston, MA  

  • OpenStack and Kubernetes. Combining the best of both worlds – Kubernetes Day

Interop ITX : May 15 – 19, 2017 in Las Vegas, NV during    Open Source IT Summit – Tuesday, May 16, 9:00 – 5:00pm  

  • 3:15 – 4:05pm OpenStack and Kubernetes
  • 4:05 – 5:00pm Kubernetes for All

Gluecon : May 24 – 25, 2017 in Denver, CO

  • Surviving Day 2 in Open Source Hybrid Automation – May 23, 2017 : Rob Hirschfeld and Greg Althaus

OTHER NEWSLETTERS

SRE Weekly (@SREWeekly)Issue #70

1 comment on “LinuxKit and Three Concerns with Physical Provisioning of Immutable Images”

LinuxKit and Three Concerns with Physical Provisioning of Immutable Images

DR ProvisionAt Dockercon this week, Docker announced an immutable operating system called LinuxKit which is powered by a Packer-like utility called Moby that RackN CTO, Greg Althaus, explains in the video below.

For additional conference notes, check out Rob Hirschfeld’s Dockercon retro blog post.

Three Concerns with Immutable O/S on Physical

With a mix of excitement and apprehension, the RackN team has been watching physical deployment of immutable operating systems like CoreOS Container Linux and RancherOS.  Overall, we like the idea of a small locked (aka immutable) in-memory image for servers; however, the concept does not map perfectly to hardware.

Note: if you want to provision these operating systems in a production way, we can help you!

These operating systems work on a “less is more” approach that strips everything out of the images to make them small and secure.  

This is great for cloud-first approaches where VM size has a material impact in cost.  It’s particularly matched for container platforms where VMs are constantly being created and destroyed.  In these cases, the immutable image is easy to update and saves money.

So, why does that not work as well on physical?

First:  HA DHCP?!  It’s not as great a map for physical systems where operating system overhead is pretty minimal.  The model requires orchestrated rebooting of your hardware.  It also means that you need a highly available (HA) PXE Provisioning infrastructure (like we’re building with Digital Rebar).

Second: Configuration. That means that they must rely on having cloud-init injected configuration.  In a physical environment, there is no way to create cloud-init like injections without integrating with the kickstart systems (a feature of Digital Rebar Provision).  Further, hardware has a lot more configuration options (like hard drives and network interfaces) than VMs.  That means that we need a robust and system-by-system way to manage these configurations.

Third:  No SSH.  Yes another problem with these minimal images is that they are supposed to eliminate SSH.   Ideally, their image and configuration provides everything required to run the image without additional administration.  Unfortunately, many applications assume post-boot configuration.  That means that people often re-enable SSH to use tools like Ansible.  If it did not conflict with the very nature of the “do-not configure-the-server” immutable model, I would suggest that SSH is a perfectly reasonable requirement for operators running physical infrastructure.

In Summary, even with those issues, we are excited about the positive impact this immutable approach can have on data center operations.

With tooling like Digital Rebar, it’s possible to manage the issues above.  If this appeals to you, let us know!

0 comments on “April 21 – Weekly Recap of All Things Site Reliability Engineering (SRE)”

April 21 – Weekly Recap of All Things Site Reliability Engineering (SRE)

Welcome to the weekly post of the RackN blog recap of all things SRE. If you have any ideas for this recap or would like to include content please contact us at info@rackn.com or tweet Rob (@zehicle) or RackN (@rackngo)

SRE Items of the Week

DigitalRebar Provision deploy Docker’s LinuxKit Kubernetes


_____________

Install Digital Rebar PXE Provision on a Mac OSX System and Test Boot using Virtual Box


_____________

Packet Pushers 333 Automation & Orchestration in Networking
http://packetpushers.net/podcast/podcasts/show-333-orchestration-vs-automation/

While the discussion is all about NETWORK DevOps, they do a good job of decrying WHY current state of system orchestration is so sad – in a word: heterogeneity.  It’s not going away because the alternative is lock-in.  They also do a good job of describing the difference between automation and orchestration; however, I think there’s a middle tier  of resource “scheduling” that better describes OpenStack and Kubernetes.

Around 5:00 minutes into the podcast, they effectively describe the composable design of Digital Rebar and the rationale for the way that we’ve abstracted interfaces for automation.  If you guys really do want to cash in by consulting with it (at 10 minutes), just contact Rob H.
_____________

Digital Magazine Launch: Increment On-Call
https://increment.com/on-call/

Increment is dedicated to covering how teams build and operate software systems at scale, one issue at a time. In this, our inaugural issue, we focus on industry best practices around on-call and incident response.
_____________

Need PXW? Try out this Cobbler Replacement
https://robhirschfeld.com/2017/04/11/provision-preview/ 

INTRO
We wanted to make open basic provisioning API-driven, secure, scalable and fast.  So we carved out the Provision & DHCP services as a stand alone unit from the larger open Digital Rebar project.  While this Golang service lacks orchestration, this complete service is part of Digital Rebar infrastructure and supports the discovery boot process, templating, security and extensive image library (Linux, ESX, Windows, … ) from the main project.

TL;DR: FIVE MINUTES TO REPLACE COBBLER?  YES.

The project APIs and CLIs are complete for all provisioning functions with good Swagger definitions and docs.  After all, it’s third generation capability from the Digital Rebar project.  The integrated UX is still evolving.
_____________

UPCOMING EVENTS

Rob Hirschfeld and Greg Althaus are preparing for a series of upcoming events where they are speaking or just attending. If you are interested in meeting with them at these events please email info@rackn.com.

DevOpsDays Austin : May 4-5, 2017 in Austin TX  

OpenStack Summit : May 8 – 11, 2017 in Boston, MA  

  • OpenStack and Kubernetes. Combining the best of both worlds – Kubernetes Day  

Interop ITX : May 15 – 19, 2017 in Las Vegas, NV

Gluecon : May 24 – 25, 2017 in Denver, CO

  • Surviving Day 2 in Open Source Hybrid Automation – May 23, 2017 : Rob Hirschfeld and Greg Althaus

OTHER NEWSLETTERS

SRE Weekly (@SREWeekly)Issue #68

2 comments on “Accelerating Community Ops on Kubernetes in Hybrid Style”

Accelerating Community Ops on Kubernetes in Hybrid Style

Preface: RackN is looking for SRE teams who are enthusiastic about accelerating Kubernetes on-premises in a long term operational way that can be shared and reused across the community.

kubernetesWe’re excited to see and be part of the community progress towards enterprise-ready Kubernetes operations on both cloud and on-premises.  The RackN team is excited to be part of multiple groups establishing patterns with shareable/reusable automation. I strongly recommend watching (or, better, collaborating in) these efforts if you are deploying Kubernetes even at experimental scale.

We’ve worked hard to make shared community ops work accessible, repeatable and multi-platform without compromising scale or security.

The RackN team has been enthusiastic supporters of Kubernetes since the 1.0 launch with our first deployments going back to June 2015 with updates for 1.2, 1.3 and now 1.5. I’m excited to report that fully converged the composable Digital Rebar approach with the Kubernetes Kargo Ansible. Our 1.2 efforts leveraged the Kargo predecessor “Kubespray.” This integration brings the parallel hybrid operation and node-by-node function of Digital Rebar with the Ansible community efforts around Kargo.

Composable design is a key element the RackN focus on SRE automation because it allows ecosystem

That allows a fully integrated deploy where Digital Rebar stages the environment and then use Kargo directly from upsteam to install Kubernetes. Post-deployment, Digital Rebar is able to extend the cluster with packages like Helm, Deis, Dashboard and others.

Since Digital Rebar supports parallel deployments, it’s possible to fully exercise the options enabled by Kargo simultaneously for development and testing.  Benefits????

For example, you can built-test-destroy coordinated Kubernetes installs on Centos, Redhat and Ubuntu as part of an automation pipeline. Unlike client side approaches like Terraform or Ansible, our infrastructure allows transparent monitoring of the deployments including Slack integration.

Flexibility is also important between users because Ops variation is both a benefit and a cost.

A key Digital Rebar design goal is for users to explore useful variation and still share operational best practices. We are proving that shared community automation can support many different scenarios including variation between between clouds, physical, operating system, networking and container engine.

If we cannot manage this variation in a consistent way then we’re doomed to operational fragmentation (like OpenStack has endured).

We’re inviting you to check out our open work supporting the Kubernetes Ops community. As Rob Hirschfeld says, looking for “Day 2” minded operators who want to make sure that we are always able to share Kubernetes best practices.

0 comments on “Why is RackN advancing OpenStack on Kubernetes?”

Why is RackN advancing OpenStack on Kubernetes?

Yesterday, RackN CEO, Rob Hirschfeld, described the remarkable progress in OpenStack on Kubernetes using Helm (article link).  Until now, RackN had not been willing to officially support OpenStack deployments; however, we now believe that this approach is a game changer for OpenStack operators even if they are not actively looking at Kubernetes.

We are looking for companies that want to join in this work and fast-track it into production. If this is interesting, please contact us at sre@rackn.com.

Why should you sponsor? Current OpenStack operators facing “fork-lift upgrades” should want to find a path like this one that ensures future upgrades are baked into the plan. This approach provide a fast track to a general purpose, enterprise grade, upgradable Kubernetes infrastructure.

Here is Rob’s Demo

Rob’s Original Blog Post

RackN revisits OpenStack deployments with an eye on ongoing operations. I’ve been an outspoken skeptic of a Joint OpenStack Kubernetes Environment because I felt that the technical hurdles of cloud native architecture would prove challenging.

I was wrong: I underestimated how fast these issues could be addressed.

… read the rest at Beyond Expectations: OpenStack via Kubernetes Helm (Fully Automated with Digital Rebar) — Rob Hirschfeld

1 comment on “Shouldn’t we have Standard Automation for Commodity Infrastructure?”

Shouldn’t we have Standard Automation for Commodity Infrastructure?

sre-seriesOur focus on SRE series continues… At RackN, we see a coming infrastructure explosion in both complexity and scale. Unless our industry radically rethinks operational processes, current backlogs will escalate and stability, security and sharing will suffer.

bookAn entire chapter of the Google SRE book was dedicated to the benefits of improving data center provisioning via automation; however, the description was abstract with a focus on the importance of validation testing and self-healing. That lack of detail is not surprising: Google’s infrastructure automation is highly specialized and considered a competitive advantage.

Shouldn’t everyone be able to do this?

After all, data centers are built from the same basic components with the same protocols.

Unfortunately, the stack of small (but critical) variations between these components makes it very difficult to build a universal solution. Reasonable variations like hardware configuration, vendor out-of-band management protocol, operating system, support systems and networking topologies add up quickly. Even Google, with their tremendous SRE talent and time investments, only built a solution for their specific needs.

To handle this variation, our SRE teams bake assumptions about their infrastructure directly into their automation. That’s expedient because there’s generally little operational reward for creating generic solutions for specific problems. I see this all the time in data centers that have server naming conventions and IP address schemes that are the automation glue between their tools and processes. While this may be a practical tactic for integration, it is fragile and site specific.

Hard coding your operational environment into automation has serious downsides.

First, it creates operational debt [reference] just like hard coding values in regular development. Please don’t mistake this as a call for yak shaving provisioning scripts into open ended models! There’s a happy medium where the scripts can be robust about infrastructure like ips, NIC ordering, system names and operating system behavior without compromising readability and development time.

Second, it eliminates reuse because code that works in one place must be forked (or copied) to be used again.  Forking creates a proliferation of truth and technical debt.  Unlike a shared script, the forked scripts do not benefit from mutual improvements.  This is true for both internal use and when external communities advance.  I have seen many cases where a company’s decision to fork away from open source code to “adjust it for their needs” cause them to forever lose the benefits accrued in the upstream community.

Consequently, Ops debt is quickly created when these infrastructure specific items are coded into the scripts because you have to touch a lot of code to make small changes. You also end up with hidden dependencies

However, until recently, we have not given SRE teams an alternative to site customization.

Of course, the alternative requires some additional investment up front.  Hard coding and forking are faster out of the gate; however, the SRE mandate is to aggressively reduce ongoing maintenance tasks wherever possible.  When core automation is site customized, Ops loses the benefits of reuse both internally and externally.

That’s why we believe SRE teams work to reuse automation whenever possible.

rebar-1Digital Rebar was built from our frustration watching the OpenStack community struggle with exactly this lesson.  We felt that having a platform for sharing code was essential; however, we also observed that differences between sites made it impossible to share code.  Our solution was to isolate those changes into composable units.  That isolation allowed us take a system integration view that did not break when inevitable changes were introduced.

If you are interested in breaking out of the script customization death spiral then review what the RackN team has done with Digital Rebar.

Even if you don’t use the code, the approach could save your SRE team a lot of heartburn down the road.  Of course, if you do want to use it then just contact us at sre@rackn.com.

0 comments on “Surgical Ansible & Script Injections before, during or after deployment”

Surgical Ansible & Script Injections before, during or after deployment

RackN CEO, Rob Hirschfeld, has been posting about our unique composable operations approach with Digital Rebar to enable hybrid infrastructure and mix-and-match underlay tooling.

This post shows some remarkable flexibility enabled by the approach that allow operators to take limited, secure operations against running systems.

via Surgical Ansible & Script Injections before, during or after deployment. — Rob Hirschfeld

 

0 comments on “Digital Rebar Training Videos”

Digital Rebar Training Videos

We’re excited to announce an updated set of Digital Rebar training videos.  In response to requests to go beyond the simple Quick Start guide, we created a dedicated training channel and have been producing 15 minute tutorials on a wide range of topics.

Want us to cover a topic?  Just ask us on Gitter!

 

In some cases, these videos contain information that has not made it into documentation yet.  Our documentation is open source, we’d love to incorporate your notes to help make the experience easier for the next user.

Thanks!