OpenStack Summit May 2014 Atlanta

Intel IT is on year 3 of our OpenStack journey. In this session we will share how we have optimized our DevOps approach to ensure we are running with the latest code in production with strong code review/testing, how we deliver enterprise level availablity for our end users, how we have integrated with our overall Enterprise systems such as Configuration Management/Identity/Service Management, and how OpenStack has become the single control plane for our datacenter infrastructure across multiple hypervisors and storage infrastructure solutions.
We will dive into both the underlying implementation of our global implementation, as well as how we are using higher level methods to enable our wide range of app developers.
This session will also go over how we are enabling a hybrid federated approach with OpenStack enabled public cloud environments, and the challenges we all need to address in order to make this seamless for the IT industry.
With 3 years under our belt with OpenStack, we will share the good, the bad, and the ugly on the reality of OpenStack for the Enterprise IT market, and share what our fellow IT peers can do now in order to drive for implementation in their respective companies.

Abstract
As Openstack matures and gets adopted, one of the potential challenges will be to run Openstack smoothly and efficiently. In order to achieve this, one way is to use operational data to improve ops. The amount of data that can be collected in an openstack deployment is quite large. It is important to extract operational insights from this data. An important step was the birth of Ceilometer which is a fantastic telemetry platform. As a first step, we show we can couple Ceilometer with a real-time stream mining platform and machine learning (using opensource components) to predict hotspots within a cluster among other patterns. We believe this approach will lead to innovations within the Openstack operations ecosystem.

Operating a large OpenStack installation can be a daunting task, but over time you'll find that the key metrics, problems, and warning signs (that you're probably spending your own personal time and energy watching for) aren't at all difficult to train a machine to do--especially if you want such a machine to become more accurate and more reliable over time in the flags it raises.
During the Hong Kong summit, we examined how a flow of logs, telemetry, and other performance data can be piped through various systems, and some possible open-source projects that can help.
In this session, we will explore in much more technical detail a setup in which logs and metrics become centralized, combined along the same timeline and presented intuitively for simpler visual inspection, as well as how to employ machine learning techniques to train your log collectors or other monitoring machines to look for patterns in the data. Having such tools in your environment will help prevent a barrage of alerts with no credence, help identify early warning signs and predict potential problems before they cause a degradation in performance, or--even worse--a service outage that interrupts your TV dinner.
But not only can these patterns be used for simple alerting as is often the traditional case; they are better used to take proactive, predefined actions, watch for recoveries, and send alerts only as needed.
Previous experience with machine learning is not necessary, but you should be slightly familiar with the inherent challenges in monitoring, alerting, and analysis persistent event flows from distributed systems.

The Massachusetts Open Cloud (MOC) will be a public cloud based on a new model that allows many companies and institutions to participate in its implementation and operation. MOC is bringing multiple vendors together within a single infrastructure, hence fostering collaboration and interoperability amongst them. Several industry partners have joined the effort and been instrumental in the initial implementation of MOC.

The goal for MOC is to foster interoperability and innovation amongst insterested parties and the opportunity to explore new computing models.

• This session will provide an overview of the architecture and it's evolution
  • Blueprint to build a multi-vendor cloud infrastrucutre
  • Architecture decisions and guidance
  • Considerations for scale, operations and upgrades
• Lessons learned from the initial deployment:
  • Issues with shared infrastructure and scope of trust
  • Solution for integrating existing hardware provisioning

We will also cover use cases for applications deployed on the MOC cloud, ranging from Big Data analytics, PaaS and others

With the participation of several key Boston universities such as Harvard, Boston University, MIT and North Eastern, UMass and other parthers MOC will evolve into a large scale open cloud infrastructure.

The OpenStack user committee will present the results of the OpenStack user survey performed during April 2014 along with the recent work with OpenStack operators to establish a requirements gathering process and feedback loop.

Title:
Scaling Neutron for large deployments
Summary:
In this presentation we will address the scaling issues faced at eBay’s Inc largest Openstack Deployment. eBay has multiple environments that help address

Description of the Presentation:
eBay Inc has been in the forefront of deploying Openstack in several environments that include eBay Market Places and PayPal production, development/QA environment.
In this talk we present our experience in scaling Neutron to deploy a large overlay and bridged network and the challenges faces.
In this talk we will go through our deployment patterns along with the HA considerations for the physical underlay network.
Some of the areas that we had to enhance to achieve scaling are listed below :

• Rabbit MQ scale and active/active mode
• Quantum servers behind VIP
• Multiple fault zone design
• Multiple worker thread and caching
• Scaling DHCP and DHCP-less (config drive)
• Multiple gateways and logical router placements

We will conclude with our observations and best practices recommendations.

About eBay Inc:

Since Networking is the glue that connects apps with infrastructure, it is critical to have network visibility. For example, operators would like to know whether the overlay or the underlay is at fault, among other things or where are the hotspots? As a first step, we explore how we can obtain the best network visibility and the challenges that lie in front, going forward. In order to solve the problems, we present strategies to 1) extract data from underlay (SDN controller) and the overlay (Neutron) system 2) show data mining and machine learning techniques to extract insights 3) interesting applications of the insights. We introduce the concept of network distances, a simple abstraction to expose the network metrics between any two entities of a network service. In particular we show how we can implement network distance within a SDN controller like Opendaylight.

If you are relatively new to OpenStack you will have discovered that it is a system that is designed to provide a very robust Infrastructure as a Service (IaaS) platform. Specifically this service is the abstraction, management, automation and orchestration of underlying infrastructure components – CPU, memory, disk and network.
You may have also noticed that it is (seemingly) lacking in other infrastructure support services, e.g. monitoring, backups, alerting, notifications, etc. It is intentionally myopic in this sense – dedicated to providing core IaaS functions without introducing complexity, bloat, points of failure and other risks for functions that are otherwise not germane to its primary purpose.
Its strength however lies in its design and development philosophy – a very open architecture, easy access to data and a powerful API. This allows for the dedicated IaaS functions to operate as efficiently as possible but also exposes the components required to provide other, typically “higher level”, infrastructure and support services. These data points and functions are often consumed by processes, applications and services outside of OpenStack, e.g. Ceilometer data pulled into an external chargeback or auditing tool, Nova or Cinder APIs polled to provide capacity or asset data to an enterprise CMDB, and so on.
These higher level" infrastructure support services are still required for a production environment however. Not every environment however has tier 1, enterprise class systems, with easy or readily available OpenStack integration so how do we provide these functions for our environment using standard open source tools?
In this session we are going to explore techniques to provide a basic level of infrastructure management services, e.g. monitoring, alerting, logging, reporting, etc., for your OpenStack environment using common and freely available open source tools, e.g. monit, collectd, syslog, etc.
If you are relatively new to OpenStack you will have discovered that it is a system that is designed to provide a very robust Infrastructure as a Service (IaaS) platform. Specifically this service is the abstraction, management, automation and orchestration of underlying infrastructure components – CPU, memory, disk and network.

You may have also noticed that it is (seemingly) lacking in other infrastructure support services, e.g. monitoring, backups, alerting, notifications, etc. It is intentionally myopic in this sense – dedicated to providing core IaaS functions without introducing complexity, bloat, points of failure and other risks for functions that are otherwise not germane to its primary purpose.

Its strength however lies in its design and development philosophy – a very open architecture, easy access to data and a powerful API. This allows for the dedicated IaaS functions to operate as efficiently as possible but also exposes the components required to provide other, typically “higher level”, infrastructure and support services. These data points and functions are often consumed by processes, applications and services outside of OpenStack, e.g. Ceilometer data pulled into an external chargeback or auditing tool, Nova or Cinder APIs polled to provide capacity or asset data to an enterprise CMDB, and so on.

These "higher level" infrastructure support services are still required for a production environment however. Not every environment however has tier 1, enterprise class systems, with easy or readily available OpenStack integration so how do we provide these functions for our environment using standard open source tools?

In this session we are going to explore techniques to provide a basic level of infrastructure management services, e.g. monitoring, alerting, logging, reporting, etc., for your OpenStack environment using common and freely available open source tools, e.g. monit, collectd, syslog, etc.
"

To promote innovation within IBM, the presenters set out to create the CloudFirst Factory, a one of a kind large multi-cloud OpenStack-based environment . Running on a total of 168 physical servers, the clouds received a substantial amount of use from the Cloud Foundry-based IBM BlueMix. The presenters will describe their experiences operating the environments with OpenStack Folsom and Grizzly releases as well as a production Folsom to Grizzly upgrade. This session also covers initial architecture decisions, the evolution of the same and the challenges in overcoming shortcomings in the original deployment.

Deploying OpenStack in a Multi-Hypervisor Enterprise Environment
Most OpenStack distributions are centered around a single hypervisor, KVM. Enterprises, on the other hand, have massive investments in a variety of different virtualization platforms. Building a private cloud is a massive change in and of itself, and most large enterprises are reluctant to lose their existing competence with their chosen hypervisor at the same time.
Learn how to integrate VMware, Hyper-V, and Xen, along with KVM into a single, heterogeneous OpenStack environment. We will also discuss various ways that a multi-hypervisor setup can help an enterprise maximize cost efficiencies, leverage existing skill sets, and expand the reach of automated services to more groups in a large enterprise.
Most OpenStack distributions are centered around a single hypervisor, KVM. Enterprises, on the other hand, have massive investments in a variety of different virtualization platforms. Building a private cloud is a severe change in and of itself, and most large enterprises are reluctant to lose their existing competence with their chosen hypervisor at the same time.
Learn how to integrate VMware, Hyper-V, and Xen, along with KVM into a single, heterogeneous OpenStack environment. We will also discuss various ways that a multi-hypervisor setup can help an enterprise maximize cost efficiencies, leverage existing skill sets, and expand the reach of automated services to more groups in a large enterprise.

This session is a panel discussion of OpenStack deployers and developers building scalable storage on top of Ceph and deploying it into production OpenStack clouds. The panel will be asked to discuss specific technical and scaling challenges faced and how to go about solving them.

With the growing popularity and footprint of IaaS platforms, the question of migration of computational workloads to these virtualized environments becomes more and more relevant. Savanna provides integration and automation for Hadoop deployment on OpenStack, but what about performance in this environment?

In this talk, we will discuss the performance impact of running Hadoop in a virtualized environment inside an OpenStack cloud.

What is the performance difference between bare-metal and virtualized Hadoop environments? What is the IO overhead in OpenStack? How does data locality influence data flow? This presentation will cover these and other questions about the performance of Hadoop on OpenStack by presenting a series of benchmark tests over different installations of Hadoop.

As we managed internal and external OpenStack deployments in IBM Labs we experimented with various architectures and approaches that improved OpenStack’s availability and performance. In this session we will discuss our use of the following technologies that helped us optimize our environments:

• MySQL Clustering
  
  • Galera (Active/Active)
  • Corosync & Pacemaker (Active/Standby)
• HAProxy and KeepAlive
• RabbitMQ Clusters

Using practical examples, we will show how we scaled out OpenStack Services (Nova, Horizon, Keystone, Glance, Cinder, etc.) and some of the challenges we ran into along the way.

In addition we will also describe a few scenarios in which we have orchestrated highly available OpenStack deployments using Heat and Chef.

Finally we will demonstrate failover and availability of the OpenStack services in an actual production cloud environment.

Like many open-source projects, OpenStack uses a time-based release model. Feature development happens in a flurry of activity in the first part of each release cycle and then we taper, allowing enough time for many eyes to make shallow bugs" and for us to gain enough confidence in the quality of our release candidates. Even after a release, we continue to maintain a stable branch for issues found and fixed post-release.

Linux distros apply a similar model when they bring together the releases of many open-source projects into a coherent, usable and stable product. OpenStack's predictable release schedule, the tapering, the stabilization and the post-release maintenance are all essential to the needs of distros and, in turn, the users of those distros.

Unusually, OpenStack also explicitly caters to another type of users - what we call "trunk chasers". Public cloud providers like HP and Rackspace invest significantly in a continuous delivery pipeline so that they can keep pace with OpenStack development, deploy regularly, give timely feedback upstream and minimize the risk associated with each incremental update. In these days of agile development, DevOps and continuous delivery, the benefits of such a model are now clear.

As such, OpenStack is an important case study for how recent thinking around continuous delivery is influencing open-source projects. Many questions remain unanswered, however. Are we going to continue to see projects like OpenStack see themselves as catering to two radically different audiences, or is this the beginning of fundamental shift in open-source?

Mark and Monty - Openstack Technical Committee and Foundation Board members, prominent OpenStack contributors and senior engineering leaders at Red Hat and HP - have had the opportunity to look at this dilemma from several different angles. In the talk, they will delve into some of the details behind how OpenStack caters to both models. They will examine the mindset and needs of each audience. They will talk about topics such as CI, upgrades, deployment tools, reference architectures, community management, feature development, user feedback and more.

Attendees can hope to learn some more about OpenStack and some of the challenges in running an open-source project, building a distro or maintaining a public cloud. Beyond OpenStack, however, the talk should provide some more general food for thought around the agile development methodologies used by many application developers today versus the methodologies used by open-source projects today.
"

At the Portland and Hong Kong Summits, Rackspace invited the OpenStack community into the their experiences deploying OpenStack trunk to their their Public Cloud Infrastructure. In this presentation, Rackspace's Deployment System Team will provide an update on the latest challenges, triumphs, and lessons learned deploying and operating a production OpenStack public cloud during the Icehouse cycle. We’ll conclude by sharing the vision for our next steps in OpenStack deployments during the Juno cycle and beyond.

Go Daddy has a large Active Directory (AD) deployment that serves many purposes for our corporate and hosting infrastructure. In building our private cloud, we've leveraged AD to power Keystone's identity service, VM authentication, and security discovery and auditing. In this talk, we'll discuss exactly how we've configured OpenStack to work well with AD, other open source tools we've used to achieve our desired functionality, and the lessons we've learned along the way. Specifically, we'll cover:

• Keystone's LDAP capabilities
  • Using LDAP for OpenStack authentication
  • Using LDAP as a store for projects and roles (and why we chose not to use this feature)
• The quirky differences between AD and LDAP and how those can impact your Keystone configuration, including known outstanding bugs (as of Havana) related to AD integration and their workarounds
• How we used the open source tool PowerBroker Identity Services to back VM authentication and possible alternative solutions
• Techniques we've used to maintain and scale the relationship between OpenStack and AD
• Using AD groups cohesively across our company's platform including: OpenStack, GitHub, CI/CD, Finance, and more

Trove is the officially incubated DBaaS (Database as a Service) project within Openstack. It's gaining momentum in the community, with contributions led by Rackspace, HP, Ebay, Mirantis, and several others. Trove is in production at most of these companies, and is the technology behind HP Cloud's DBaaS offering.

In this session, we'll talk about

• How to run Trove in a fully-HA mode,
• Trove configuration management based around SaltStack
• Security and Monitoring of key Trove components
• Upgrading Service
• How we operate the service in the public cloud

Introduction to OpenStack Trove: A multi-database deployment with MySQL and MongoDB
Co-speakers Michael Basnight, PTL of Trove, and Doug Shelley, active developer on Trove
A key goal of Trove is to provide a scalable and reliable DBaaS provisioning functionality for both relational and non-relational database engines. This tutorial covers the steps involved in a simple multi-database deployment of OpenStack Trove with MySQL and MongoDB.
We begin with a short introduction to Trove, describe the steps involved in installing Trove and a brief description of the Trove API. We then demonstrate some interactions with the Trove API including a demonstration of a deployment with a relational database (MySQL) and a non-relational database (MongoDB).
We conclude with a brief description of emerging trends in the Database-as-a-Service and Data-as-a-Service landscape.
During this session you will learn about:

• What is Trove
• What is the Trove API and how can I interact with Trove
• How do I install Trove
• How can I deploy different data stores with Trove
• The future of DBaaS and DaaS

OpenStack is complex. Arguably one of the more complex
cloud platforms available. As its scope continues to grow,
so does the complexity. OpenStack is considered difficult
to deploy, operate, and configure.

There is no shortfall of communities, projects, and products
aiming to solve these problems, and I have worked with many
of them. Currently I am involved in the Chef OpenStack
community, but have extensive experience operating customer
owned OpenStack deployments with Ansible.

In this talk I will describe ways to leverage Ansible to
tackle OpenStack deployment, orchestration, orchestration
of other config management systems, day-to-day/ad-hoc
operations, and integration testing. 

Continuous Integration / Continuous Delivery (CI/CD) is a well established practice in the software engineering of applications. With the advent of the cloud and OpenStack, new challenges have arisen in how best to apply those patterns to the infrastructure upon which those applications are running. Add the unique challenges posed by consuming an open source code base, and the opportunities for innovation and creative solutions are great. In this talk, members of Rackspace's Deployment System Team will cover how they have adapted CI/CD patterns and principles to maintain the infrastructure of their public cloud offering (compute, networks, images.) The discussion will offer a look into the evolution of the deployment process of a large-scale OpenStack cloud, as well as the tools used to achieve the current level of automation around the consumption and deployment of upstream OpenStack code. The talk will conclude with an overview of ongoing efforts to continually improve and automate the interaction between the physical capacity of the cloud and the software bits that enable virtual instances to be created.

Target audience:
Administrators of VMware environments looking to leverage Openstack
Cloud builders researching compute platforms
Technical decision makers on Openstack components
People looking to gain knowledge on what VMware products can bring in an Openstack environment
Business people looking to leverage existing investments


Speakers:
Justin Giardina - CTO, iland Internet Solutions
Julien Angeunot - Director of Software Engineering, iland Internet Solutions

In this session, we will take a deep look into what a VMware vSphere backend can bring to an Openstack deployment. We will cover in detail:
Relationships between the Openstack components and how they relate to a VMware vSphere environment (Nova, Glance, Neutron, etc.).
The power of vSphere for backend operations such as performance, management, networking, high availability, etc.
Implementing VMware's advanced L2-L7 network features to provide services such as firewalling, load balancing, NAT, VPN, etc. in Openstack.
Using Openstack on top of an existing or purpose-built VMware infrastructure
Example architectures to create a fault tolerant and fully software defined Openstack cloud with VMware vSphere.

Even the smallest cloud has the potential to generate huge amounts of log information. Much of it is redundant, awkwardly formatted, and generally of little value, but hidden in there are clues that can help you understand that you are having, or more importantly, about to have a problem. Logs are an important triage tool to help you get out of the woods.

This presentation discusses some of the experiences, challenges, and revelations had while deploying a log management infrastructure for Openstack. It will cover topics such as handling large volumes of logs, filtering and parsing of different types of events, and some interesting queries that can help. Finally, it will cover some suggested improvements that the Openstack community might consider to make the process of log ingestion easier and log mining more fruitful.

One of the tricker parts of productionalizing a modern OpenStack cloud is how to minimize downtime at the tenant logical router level. Much progress is being made, but the grizzly and havana codebases are still heavily used and have very little mindshare surrounding how to best provide a highly available logical router agent environment.

In this talk I will discuss some of the strategies I have implemented, solicit suggestions or insight from others, and dive into what I believe are the biggest hurdles keeping stock L3 agent logical routers from being truly highly available.

Integration for Docker in OpenStack was introduced with Havana with plugins for OpenStack Compute (Nova) and Heat.
In this talk, we'll introduce you to Docker, provide examples for using Heat, and explain what changes are necessary when you deploy Nova with Docker. Finally, we'll explain some best practices to ensure you deploy Docker integrated OpenStack in a secure and scalable fashion.