Category: Uncategorized

Digital Rebar Platform v4.2 Release

Posted by Rob Hon in Uncategorized

These release notes span the v4.1 and v4.2 releases. The focus is on features, not the numerous fixes for bugs and minor issues.

4.1 Release Items

  • Contexts – this is a major feature that allows workflows to transition between instances of runners on the machines and other locations (such as Docker containers)
  • Run without root permissions – enables operation in highly secure environments.
  • Tasks can have expanded Role and Claim rights – allows tasks to have special scope of operations inside of DRP
  • Transactions Improvements to WAL 
  • DRPY runner –  python version of the runner for ESXi environments
  • Ansible Inventory uses Profiles – 100% mapping between DRP and Ansible groups.
  • Redfish Support – IPMI integration for later hardware models
  • ESXi Installation – VMware Integration

    UX Custom Columns – Allows user to define columns in machine view

  • UX Expand View Filters – Expand filters to leverage all API supported filtering

4.2 Release Items

  • JQ integrated into DRP CLI – eliminates the need for installing JQ as a stand alone requirement.
  • BSDTAR integrated into DRP Server – eliminates the need for installing bsdtar on the server
  • Human/table formats for DRPCLI – makes it easier to read DRP output
  • DRP CLI can install itself as a persistent agent – from script contexts makes it easier to run DRP runner as a service
  • Plugins use Websocket events – migrate for legacy event model to improve stability.
  • UX Save in Header – makes it easier to have multiple edits on the same item

    UX diff view – allow operator to compare new vs installed content (fixes regression)

  • Updated license retrieval and install process – streamline process so that multiple users do not need RackN logins if a license is installed
  • UX bulk actions for Endpoints – useful for multi-site management

Beta Items

  • Bootstrap Agent Integrated into DRP Server – allows for on DRP server operations without use of Dangerzone context (v4.2)

 

Server firmware is fragile! Why haven’t we fixed it? The HPE SSD patch challenge.

Posted by Rob Hon in Uncategorized
Clocks
https://www.pexels.com/photo/art-city-clock-clock-face-277458/

Last week, HPE alerted us of a critical software bug that would cause SSDs to fail after three point seven years of use. The problem is not that there was a firmware bug, those happen all the time, but how little HPE provided to fix the problem. It basically throws the fix back to the operator to run scripts on each server.

SSDs are an example of firmware that is not typically managed by vendor out of band (HPE iLO, Dell DRAC) tooling. PDU, Fan, GPU, FPGA, and NIC firmware are also examples of deep firmware that must also be managed.

When we formed RackN 5 years ago, we thought that operators would want to make sure their server firmware was patched; however, in spite of a string of critical patches, firmware continually falls under the threshold of must manage software. This is remarkable because it is a well known performance and failure point for infrastructure.

We’ve spent the last few years trying to understand why the hardware is so often ignored in the infrastructure equation and refuse to accept that “it’s too hard” as the answer.

Instead, our industry has failed to treat firmware as part of an integrated infrastructure stack. We’ve accepted poor APIs and controls from vendors. We’ve fostered a belief that virtualization shields us from hardware. We’ve accepted the cloud “your not smart enough to run a data center” marketing. We’ve allowed these fallacies to make us complacent.

We need to treat firmware patching as part of a continuously integrated data center mindset. Where we can perform source-code controlled migrations of systems that allow us to safely and quickly make deep changes as part of regular deployment cycles.

Fully integrated provisioning is not fantasy: At RackN, we’re helping operators implement exactly these processes. It’s not magic, but it does require system level processes that prioritize workflows, inventory, discovery and integration to coordinate physical and application lifecycle.  It’s time for everyone to start thinking this way because this SSD patch is just the latest in an increasingly urgent string of firmware updates.

YES – VM + Containers can be faster than Bare Metal!

Posted by Rob Hon in Uncategorized1 Comment

Pat Gelsinger, VMware CEO, said that VM managed Containers could be 8% faster than bare metal during the VMworld keynote (@kitcolbert). On the surface, this comment defies logic: bare metal should be the theoretical limit for performance like the speed of light. While I don’t know the specifics of his test case, the claim of improved performance is credible.  Let’s explore how.

RackN specializes in bare metal workloads so let me explain how it’s possible in the right cases that containers in VMs benchmark faster than containers alone.

The crux of the argument comes down to two factors:

  1. Operating systems degrade when key resources are depleted
  2. CPUs are optimized for virtualization (see NUMA architecture)

Together, these factors conspire to make VMs a design necessity on large bare metal systems.

A large RAM and CPU core system can become saturated with container workloads even in the 10s of containers. In these cases, the performance cost for operating system to manage resources starts to take away from the performance. Since typical hypervisor hosts have a lot of resources, the risk of over saturation is very high.

The solution on high resource hosts is to leverage a hypervisor to partition the resources into multiple operating system instances. That eliminates over saturation and improves throughput for the host. We’re talking about 10 vms with 10 containers instead of 1 host with 100 containers.

In addition to simple partitioning, most CPUs are optimized for virtualization. That means that they can run multiple virtualization operating systems on the same host with minimal overhead.  The non-virtualized host does not get to leverage this optimization.

Due to these factors AND with the right tuning, it would be possible to demonstrate improved container performance for hosts that were optimized for running a hypervisor. The same would not hold true for systems that are size optimized for only container workloads. Since the container optimized machines are also much cheaper, the potential performance gain is likely an not a good ROI.

While bare metal will eventually come; this strange optimization reinforces why we expect to see hypervisors continue to be desired in container deployments for a long time.

RackN Digital Rebar v4 Release

Posted by Rob Hon in Uncategorized

RackN is excited to announce v4 of Digital Rebar.  In this fourth generation, we recognize the project’s evolution from provisioner into a data center automation platform.  While the external APIs and behavior are NOT changing (v4.0 and v3.13 are API compatible), we are making important structural changes that allow for significant enhancements to scale, performance and collaboration.

This important restructuring enables Digital Rebar to expand as a community platform with open content.

RackN is streamlining and inverting our license model (see chart below) by making our previously closed content and plugins open and taking commercial control of our implementation behind the open Digital Rebar APIs.  This change empowers both expanding community and sustaining commercial engagement.

Over the last 2 years, our operator community around Digital Rebar and RackN content has frequently asked for source access to our closed catalog of content and plugins while leaving core development exclusively to RackN.  We believe that providing open, collaborative access is critical for the growth of the platform; consequently, Generation 4 makes these critical components available.

RackN will curate and support foundational content packs and plugins in the open.  They include items including operating system templates, out of band management (IPMI), image deploy, classification and our server life-cycle (RAID/BIOS) components.  We will be working with the Digital Rebar community to determine the right management model for items in this broad portfolio of content.

The RackN implementation powering the Digital Rebar API will be significantly enhanced in this generation.

We are integrating powerful new capabilities like multi-site management, 10,000+ machine scale, single-sign-on and deeper security directly into the implementation.  These specialized enhancements to the platform enable an even broader range of applications. Since the community has focused on contributing to platform content instead of the core; we believe this is the right place to manage the commercial side of sustaining the platform.

Please note that our basic commercial models and pricing plans are not changing.  We are still maintaining Digital Rebar and RackN binaries free for operators with up to 20 machines.  The change also allows us to enable new classes of use such as non-profit, academic and service provider.

Thank you to the Digital Rebar community!  We are handing you the keys to revolutionized data center operations.  Let’s get started!

Generation 4 clarifies the boundary between open and proprietary
Layer Component Generation 3 Generation 4
Content Platforms, Apps, Etc Mixed Digital Rebar APLv2
Ops Best Practices RackN Digital Rebar APLv2
O/S Image Deployer RackN Digital Rebar APLv2
OOB & Firmware RackN Digital Rebar APLv2
Utility Workflows RackN Digital Rebar APLv2
O/S Install Templates Mixed Digital Rebar APLv2
Platform Agents (Runner) Digital Rebar APLv2 Digital Rebar APLv2
CLI Digital Rebar APLv2 Digital Rebar APLv2
Sledgehammer Digital Rebar APLv2 Digital Rebar APLv2
API Digital Rebar APLv2 Digital Rebar APLv2
Enterprise Extensions (Multi-Site, SSO, RBAC) RackN RackN
Web UX RackN RackN
API Implementation Digital Rebar APLv2 RackN
Commercial Support RackN RackN

RackN founding member of LF Edge

Posted by Rob Hon in Uncategorized

Today, the Linux Foundation announced LF Edge to help build open source platforms and community around Edge Infrastructure. We’re just at the start of the effort with projects and vendors working in parallel without coordination. We think this LF Edge initiative is important because the Edge is already a patchwork of different data centers, infrastructure and platforms.

RackN has been designing our DEEP Edge federated control around open Digital Rebar because we know that having a shared abstraction at the physical layer makes everything we build above easier to sustain and more resilient.

We’re excited to be part of this effort and help that you’ll collaborate us to build some amazing Edge capabilities.

Note: We’re also watching the OpenStack Foundation work with Edge also. Hopefully, we’ll see collaboration between these groups since there are already overlapping projects.

Podcast – Ian Rae talks Cloud, Innovation, and Updates from Google Next 2018

Posted by Rob Hon in Kubernetes, OpenStack, UncategorizedLeave a comment

Joining us this week is Ian Rae, CEO and Founder CloudOps who recorded the podcast during the Google Next conference in 2018.

Highlights

  • 1 min 55 sec: Define Cloud from a CloudOps perspective
    • Business Model and an Operations Model
  • 3 min 59 sec: Update from Google Next 2018 event
    • Google is the “Engineer’s Cloud”
    • Google’s approach vs Amazon approach in feature design/release
  • 9 min 55 sec: Early Amazon ~ no easy button
    • Amazon educated the market as industry leader
  • 12 min04 sec: What is the state of Hybrid? Do we need it?
    • Complexity of systems leads to private, public as well as multiple cloud providers
    • Open source enabled workloads to run on various clouds even if the cloud was not designed to support a type of workload
    • Google’s strategy is around open source in the cloud
  • 14 min 12 sec: IBM visibility in open source and cloud market
    • Didn’t build cloud services (e.g. open a ticket to remap a VLAN)
  • 16 min 40 sec: OpenStack tied to compete on service components
    • Couldn’t compete without Product Managers to guide developers
    • Missed last mile between technology and customer
    • Didn’t want to take on the operational aspects of the customer
  • 19 min 31 sec: Is innovation driven from listening to customers vs developers doing what they think is best?
    • OpenStack is seen as legacy as customers look for Cloud Native Infrastructure
    • OpenStack vs Kubernetes install time significance
  • 22 min 44 sec: Google announcement of GKE for on-premises infrastructure
    • Not really On-premise; more like Platform9 for OpenStack
    • GKE solve end user experience and operational challenges to deliver it
  • 26 min 07 sec: Edge IT replaces what is On-Premises IT
    • Bullish on the future with Edge computing
    • 27 min 27 sec: Who delivers control plane for edge?
      • Recommends Open Source in control plan
  • 28 min 29 sec: Current tech hides the infrastructure problems
    • Someone still has to deal with the physical hardware
  • 30 min 53 sec: Commercial driver for rapid Edge adoption
  • 32 min 20 sec: CloudOps building software / next generation of BSS or OSS for telco
    • Meet the needs of the cloud provider for flexibility in generating services with the ability to change the service backend provider
    • Amazon is the new Win32
  • 38 min 07 sec: Can customers install their own software? Will people buy software anymore?
    • Compare payment models from Salesforce and Slack
    • Google allowing customers to run their technology themselves of allow Google to manage it for you
  • 40 min 43 sec: Wrap-Up

Podcast Guest: Ian Rae, CEO and Founder CloudOps

Ian Rae is the founder and CEO of CloudOps, a cloud computing consulting firm that provides multi-cloud solutions for software companies, enterprises and telecommunications providers. Ian is also the founder of cloud.ca, a Canadian cloud infrastructure as a service (IaaS) focused on data residency, privacy and security requirements. He is a partner at Year One Labs, a lean startup incubator, and is the founder of the Centre cloud.ca in Montreal. Prior to clouds, Ian was responsible for engineering at Coradiant, a leader in application performance management.

Podcast – Yves Boudreau on State of Edge Report and Edge vs Cloud

Posted by Rob Hon in Open Source, Podcast, UncategorizedLeave a comment

Joining us this week is Yves Boudreau from Ericsson for his 2nd Podcast appearance (1st Podcast) to talk about the new State of the Edge Report and the latest happenings in the Edge community.

Highlights

  • Edge as an accelerant not having to wait until Edge is built completely
  • Opportunity Cost using Edge as is; no time to wait
  • Be Specific when Requesting Services
  • Internet and Networks are Not Unlimited Pipes
  • Interesting Use Cases for Edge – Augmented Reality, Drone, Cars, Batteries
  • Cost savings of where the data processing is done
  • Open Source software communities at the Edge

Topic                                                                                    Time (Minutes.Seconds)

Intro                                                                                             0.0 – 1.22
State of the Edge Report                                                         1.22 – 5.22 (STE Podcast)  (https://www.stateoftheedge.com/)
Accessible Edge Environments                                              5.22 – 10.50 (Bulgaria)
Opportunity Cost and Missing Killer App                             10.50 – 12.04
Edge Infrastructure as Cloud Development Paradigm      12.04 – 14.29
Elasticity Issues b/w Cloud and Edge                                  14.29 – 21.45
Innovators Dilemma for Cloud & Telcom                             21.35 – 23.10
Favorite Use Cases for Infrastructure Edge                         23.10 –  28.55 (Hanger Podcast)
Data Location and Data Sovereignty                                    28.55 – 31.03
Cost for Processing Power in Edge Devices                        31.03 – 34.49 (SWIM.AI Podcast)
Free Software/ Open Source in Edge                                   34.49 – 46.58
Wrap Up                                                                                     46.58 – END

 

 

Podcast Guest:  Yves Boudreau, VP Partnership and Ecosystem Strategy

Mr. Boudreau is a 20 year veteran of the Digital, Telecom and Cable TV industries. From modest beginnings of one of the first cable broadband ISPs in Canada to the fast paced technology hub of Silicon Valley, Yves joined ERICSSON in 2011 as Vice President of Technical Sales Support and most recently has accepted a position as the VP of Partnerships and Ecosystem Strategy for the ERICSSON Unified Delivery Network. Previously, Mr. Boudreau has worked in R&D, Systems Engineering & Business Development for companies such as Com21 Inc., ARRIS Group (Cable), Imagine Communication (Video Compression) and Verivue Inc. (CDN). Yves now resides in Atlanta, Georgia with his wife Josée and 3 children. Mr. Boudreau completed his undergraduate studies in Commerce @ Laurentian University and graduate studies in Information Technology Management @ Athabasca University. Yves currently also serves on the Board of Director of the Streaming Video Alliance (www.streamingvideoalliance.org)

December 1 – Weekly Recap of Digital Rebar, RackN, and Industry News

Posted by Rob Hon in DevOps, Digital Rebar, Metal, rackn, Uncategorized, Week in ReviewLeave a comment

Welcome to the weekly post of the RackN blog recap of all things Digital Rebar, RackN, Edge Computing, and DevOps. If you have any ideas for this recap or would like to include content please contact us at info@rackn.com or tweet RackN (@rackngo)

Items of the Week

Industry News

Edge computing, in the context of IoT, is the idea that you can actually do some of the computational work required by a system close to the endpoints instead of in a cloud or a data center. The intent is to minimize latency, which, according to Renaud, means that it’s going to be a hot trend in certain kinds of industrial IoT application.

Solution providers that have been hit hard by a data center hardware retreat are finding sales and profit growth by living on the edge—the network edge, that is.

DevOps — a term used to refer to the integration of software developers and operations teams — continues to spread like wildfire throughout the open networking ecosystem. The main idea behind DevOps is that by breaking down barriers between these two departments, market applications can be delivered faster with lower costs and better quality. Nevertheless, for all the advantages attached to DevOps, it is still a budding concept since it is primarily concerned with re-aligning the workforce with a variety of tools. The following, therefore, is a list of DevOps trends to keep an eye out for.

Digital Rebar

Our architectural plans for Digital Rebar are beyond big – they are for massive distributed scale. Not up, but out. We are designing for the case where we have common automation content packages distributed over 100,000 stand-alone sites (think 5G cell towers) that are not synchronously managed. In that case, there will be version drift between the endpoints and content. For example, we may need to patch an installation script quickly over a whole fleet but want to upgrade the endpoints more slowly.

Prior Meetup on November 21st Notes

RackN

Yesterday, AWS confirmed that it actually uses physical servers to run its cloud infrastructure and, gasp, no one was surprised.  The actual news about the i3.metal instances by AWS Chief Evangelist Jeff Barr shows that bare metal is being treated as just another AMI managed instance type (see also Geekwire, Techcrunch, Venture Beat).  For AWS users, there’s no drama here because it’s an incremental add to processes they are already know well.

We are actively looking for feedback from customers and technologists before general availability of both RackN and the Terraform plug-in. It takes just a few minutes to get started and we offer direct engineering engagement on our community slack channel. Get started now by providing your email on our registration pagey so we can provide you all the necessary links.

L8ist Sh9y Podcast

Podcast Guest: Krishnan Subramanian, Rishidot Research

Founder and Chief Research Advisor, Infrastructure, Application Platforms and DevOps

UPCOMING EVENTS

  • KubeCon + CloudNativeCon : Dec 6 – 8 in Austin, TX

Event plans for the RackN and Digital Rebar team include 2 sessions and the RackN booth. We look forward to seeing you in Austin.

The RackN team is preparing for a series of upcoming events where they are speaking or just attending. If you are interested in meeting with them at these events please email info@rackn.com

 

 

 

 

 

 

Deep Thinking & Tech + Great Guests – L8ist Sh9y Podcast

Posted by Rob Hon in Digital Rebar, Kubernetes, Podcast, Technology, UncategorizedLeave a comment

I love great conversations about technology – especially ones where the answer is not very neatly settled into winners and losers (which is ALL of them in IT).  I’m excited that RackN has (re)launched the L8ist Sh9y (aka Latest Shiny) podcast around this exact theme.

Please check out the deep and thoughtful discussion I just had with Mark Thiele (notes) of Apcera where we covered Mark’s thought on why public cloud will be under 20% of IT and culture issues head on.

Spoiler: we have David Linthicum coming next, SO SUBSCRIBE.

I’ve been a guest on some great podcasts (Cloudcast, gcOnDemand, Datanauts, IBM Dojo, HPE, Foodfight) and have deep respect for critical work they do in industry.

We feel there’s still room for deep discussions specifically around automated IT Operations in cloud, data center and edge; consequently, we’re branching out to start including deep interviews in addition to our initial stable of IT Ops deep technical topics like Terraform, Edge Computing, GartnerSYM review, Kubernetes and, of course, our own Digital Rebar.

Soundcloud Subscription Information

OpenStack’s Big Pivot: our suggestion to drop everything and focus on being a Kubernetes VM management workload

Posted by Rob Hon in Kubernetes, OpenStack, Uncategorized6 Comments

TL;DR: Sometimes paradigm changes demand a rapid response and I believe unifying OpenStack services under Kubernetes has become an such an urgent priority that we must freeze all other work until this effort has been completed.

See Also Rob’s VMblog.com post How is OpenStack so dead AND yet so very alive

By design, OpenStack chose to be unopinionated about operations.

pexels-photo-422290That made sense for a multi-vendor project that was deeply integrated with the physical infrastructure and virtualization technologies.  The cost of that decision has been high for everyone because we did not converge to shared practices that would drive ease of operations, upgrade or tuning.  We ended up with waves of vendors vying to have the the fastest, simplest and openest version.  

Tragically, install became an area of competition instead an area of collaboration.

Containers and microservice architecture (as required for Kubernetes and other container schedulers) is providing an opportunity to correct this course.  The community is already moving towards containerized services with significant interest in using Kubernetes as the underlay manager for those services.  I’ve laid out the arguments for and challenges ahead of this approach in other places.  

These technical challenges involve tuning the services for cloud native configuration and immutable designs.  They include making sure the project configurations can be injected into containers securely and the infra-service communication can handle container life-cycles.  Adjacent concerns like networking and storage also have to be considered.  These are all solvable problems that can be more quickly resolved if the community acts together to target just one open underlay.

The critical fact is that the changes are manageable and unifying the solution makes the project stronger.

Using Kubernetes for OpenStack service management does not eliminate or even solve the challenges of deep integration.  OpenStack already has abstractions that manage vendor heterogeneity and those abstractions are a key value for the project.  Kubernetes solves a different problem: it manages the application services that run OpenStack with a proven, understood pattern.  By adopting this pattern fully, we finally give operators consistent, shared and open upgrade, availability and management tooling.

Having a shared, open operational model would help drive OpenStack faster.

There is a risk to this approach: driving Kubernetes as the underlay for OpenStack will force OpenStack services into a more narrow scope as an infrastructure service (aka IaaS).  This is a good thing in my opinion.   We need multiple abstractions when we build effective IT systems.  

The idea that we can build a universal single abstraction for all uses is a dangerous distraction; instead; we need to build platform layers collaborativity.  

While initially resisting, I have become enthusiatic about this approach.  RackN has been working hard on the upgradable & highly available Kubernetes on Metal prerequisite.  We’ve also created prototypes of the fully integrated stack.  We believe strongly that this work should be done as a community effort and not within a distro.

My call for a Kubernetes underlay pivot embraces that collaborative approach.  If we can keep these platforms focused on their core value then we can build bridges between what we have and our next innovation.  What do you think?  Is this a good approach?  Contact us if you’d like to work together on making this happen.

See Also Rob’s VMblog.com post How is OpenStack so dead AND yet so very alive to SREs? 

%d bloggers like this: