CoreOS Blog

Meet the CoreOS team around the world in August

2015-08-04T00:00:00+00:00

This month the CoreOS team will be speaking from locations along the Pacific Northwest in the US, to Austria, to Japan and China. August also begins our Kubernetes workshop series, brought to you by Google and Intel.

Wednesday, August 5, 2015 at 10:00 a.m. PST – Portland, OR

We kick off our Kubernetes training series in Portland with Kelsey Hightower (@kelseyhightower), product manager, developer and chief advocate at CoreOS. This hands-on workshop will teach you everything you need to know about Kubernetes, CoreOS and Docker. We are offering the workshop for only the cost of materials ($75) for a limited time so we encourage you to send any members of your team for this date. Register in advance to attend.

Friday, August 7, 2015 at 10:00 a.m. PST – Seattle, WA

Kelsey will provide the next Kubernetes training in Seattle, guiding your team through Kubernetes, CoreOS and Docker for only the cost of materials ($75) for a limited time. This event is sold out but we have several other trainings in other cities.

Friday, August 7, 2015 at 4:00 p.m. PST – Las Vegas, NV

Going to DEF CON 23 this year? Meet Brian “Redbeard” Harrington (@brianredbeard), who will speak in a Skytalk on container security and kernel namespaces on Friday, August 7.

In case you missed it, see Redbeard’s presentation on minimal containers from the CoreOS + Sysdig San Francisco July meetup.

Monday, August 10, 2015 at 10:00 a.m. PST – San Francisco, CA

Join us for a daylong Kubernetes training in San Francisco. Kelsey will walk you through Kubernetes, CoreOS and Docker. Seats are filling up quickly so register early to secure your spot.

Tuesday, August 11, 2015 at 6:30 p.m. BST – London, UK

Join Iago López (@iaguis), senior developer, for the Container Infrastructure Meetup at uSwitch in London. He’ll provide an overview and update on rkt, a container runtime designed to be composable, secure and fast.

Monday, August 17, 2015 at 2:20 p.m. PST – Seattle, WA

CoreOS will be at LinuxCon and ContainerCon for the week! Join us for a variety of talks in Seattle.

First, at 2:20 p.m. PT see Jonathan Boulle (@baronboulle), product technical lead at CoreOS, speak at ContainerCon to discuss the importance of creating a common, interoperable and well-defined specification standardizing what “application containers” are.
Immediately following, see Brandon Philips (@brandonphilips), CTO at CoreOS, speak at 3:20 p.m. PT about the Application Containers Stack, rkt to Kubernetes.
Later at 6:20 p.m. PT, Brandon Philips will join the day one closing container keynote panel with Tim Hockin (@thockin), technical lead, cluster management at Google, Matt Trifiro (@mtrifiro), SVP at Mesosphere, Jérôme Petazzoni (@jpetazzo) of Docker, and moderator Joe Brockmeier (@jzb), who works on open source and standards at Red Hat.

Wednesday, August 19, 2015 at 7:00 p.m. JST – Tokyo, Japan

Save the date! Kelsey Hightower will be speaking at a meetup in Tokyo. More details will be added – stay tuned on our community page for updates.

Wednesday, August 19, 2015 at 10:25 a.m. PST – Seattle, WA

More CoreOS talks at LinuxCon and ContainerCon include two speakers with expertise in security and networking.

First at 10:25 a.m. PT, Matthew Garrett (@mjg59), principal security software engineer at CoreOS, will present Tying TPMs Throughout The Stack.
Later at 4:00 p.m. PT, Eugene Yakubovich (@eyakubovich), senior software engineer and maintainer of flannel at CoreOS, will present From Network Namespace to Fabric Overlays.

Thursday, August 20, 2015 at 9:30 a.m. PST – Seattle, WA

From LinuxCon and ContainerCon, our team will also be speaking at Linux Plumber’s Conference in Seattle. Brandon Philips will kick off the event with a talk on Open Containers.

Friday, August 21, 2015 at 11:10 a.m. JST – Tokyo, Japan

Meet Kelsey Hightower in Tokyo at YAPC Asia. He’ll discuss managing containers at scale with CoreOS and Kubernetes.

Friday, August 21, 2015 at 9:00 a.m. PST – Seattle, WA

Linux Plumbers Conference attendees are welcome to join Matthew Garrett to learn about securing the entire boot chain.

MesosCon attendees should not miss Brandon Philips discussing rkt and more at 11:30 a.m. PT.

Tuesday, August 25, 2015 – Vienna, Austria

Jonathan Boulle will be giving a keynote at Virtualization in High-Performance Cloud Computing (VHPC ’15), held in conjunction with Euro-Par 2015, in Vienna, Austria. Jon will discuss the work behind designing an open standard for running applications in containers.

Wednesday, August 26, 2015 at 11:35 a.m. PST – Mountain View, CA

OpenStack Silicon Valley, hosted at the Computer History Museum, will feature Alex Polvi (@polvi), CEO of CoreOS. He’ll present Containers for the Enterprise: It's Not That Simple on August 26 at 11:35 a.m. PT.

Immediately following is a deep-dive session with Wall Street Journal technology reporter Shira Ovide (@ShiraOvide), joined by Alex, James Staten, chief strategist of the cloud and enterprise division at Microsoft, as well as Craig McLuckie (@cmcluck), senior product manager at Google. They will discuss practical choices facing enterprises moving to an IT resource equipped to support software developers in their work to help their companies compete.

Friday, August 28, 2015 – Beijing, China

At CNUT Con, presented by InfoQ in Beijing, Kelsey Hightower will give a keynote: From Theory to Production: Managing Applications at Scale.

To invite CoreOS to a meetup, training or conference in your area email us or tweet to us @CoreOSLinux!

Introducing etcd 2.1

2015-07-24T00:00:00+00:00

After months of focused work, etcd 2.1 has been released. Since the etcd 2.0 release in January, the team has gathered a ton of valuable feedback from real-world environments. And based on that feedback, this release introduces: authentication/authorization APIs, new metric endpoints, improved transportation stability, increased performance between etcd servers, and enhanced cluster stability.

For a quick overview, etcd is an open source, distributed, consistent key value store for shared configuration, service discovery, and scheduler coordination. By using etcd, applications can ensure that even in the face of individual servers failing, the application will continue to work. etcd is a core component of CoreOS software that facilitates safe automatic updates, coordinating work being scheduled to hosts, and setting up overlay networking for containers.

If you want to skip the talk and get right to the code, you can find new binaries on GitHub. etcd 2.1.1 is available in CoreOS 752.1.0 (currently in the alpha channel), so feel free to take it for a spin.

Zero-Downtime Rolling Upgrade from 2.0

Upgrading from etcd 2.0 to etcd 2.1 is a zero-downtime rolling upgrade. The basic approach is that you can upgrade a cluster running etcd 2.0 one-by one to etcd 2.1. For more details, please read the upgrade documentation. If you are running your cluster under etcd 0.4.x, please upgrade to etcd 2.0 first and then follow the rolling upgrade steps.

Also, with this release, etcd 2.1 is now the current stable etcd release; as such, all bug fixes will go into new etcd 2.1.x releases and won't be backported to etcd 2.0.x.

Auth API for Authentication and Authorization

A major feature in this release is the /v2/auth endpoint, which adds auth to the etcd key/value API. This API lets you manage authorization of key prefixes with users and roles and authenticate those users using HTTP basic authentication, enabling users to have more control within teams. This includes support in the etcd HTTP server, the command-line etcdctl client, and the Go etcd/client package. You can find full details in the authentication documentation. Please note that this is an experimental feature and will be improved based on user feedback. We think we got the details right but may adjust the API in a subsequent release.

Improved Transport Stability

Many users of etcd have networks with inconsistent performance and latency. We can't make etcd work perfectly in all of these difficult environments but what we have done in this release is optimize the way etcd uses the network in a variety of ways to make it perform in an optimal manner.

First, to reduce the connection creation overhead and to make the consensus protocol (raft) communication more efficient and stable, etcd now maintains long running connections with other peers. Next, to reduce the raft command commit latency, each raft append message is now attached to a commit index. The commit latency is reduced from 100ms to 1ms under light load (<100 writes/second). And finally, etcd's raft implementation now provides better internal flow control, significantly reducing the possibility of raft message loss, and improving CPU and memory efficiency.

Functional Testing

For four months we have been running etcd against a fault-injecting and functional testing framework we built. Our goal is to ensure etcd is failure-resistant while under heavy usage; and in these months of testing, etcd has shown to be robust under many kinds of harsh failure scenarios. We will continue to run these tests as we iterate on the 2.1 releases.

Improved Logging

Leveled logging is supported now. Users can set an expected log level for etcd and its subpackages. In the meantime, we have moved verbose, repeated logging to DEBUG log level, so etcd's default log will be significantly more readable. You can control leveled logging using flags listed here.

New Metrics API

etcd 2.1 introduces a new metrics API endpoint that can be used for real-time monitoring and debugging. It exposes statistics about both client behaviors and resource usage. Like the auth API endpoint, this is an experimental feature which may be improved and changed based on user feedback.

Get Involved and Get Started

We will continue to work to make etcd a fundamental building block for Google-like infrastructure that users can take off the shelf, build upon, and rely on. Get started with etcd, continue to share your feedback, and even help by contributing directly to the code.

CoreOS and Kubernetes 1.0

2015-07-21T00:00:00+00:00

Today is a big day for Kubernetes, as it hits its 1.0 release milestone. Kubernetes provides a solid foundation for running container-based infrastructure providing API driven deployment, service discovery, monitoring and load balancing. It is exciting progress towards bringing industry-wide Google-like infrastructure for everyone else (GIFEE) through community-built open source software.

Kubernetes 1.0 on CoreOS Open Source Guides

The Kubernetes project has come a long way in just over a year, with many API improvements and more recently a focus on stability and scalability. If you haven't tried Kubernetes recently it is a worthwhile experience and can get you thinking how containers can be more easily used in real-world deployments: whether it is doing your first rolling upgrade of your containerized app or using DNS service discovery between components.

For those that want to try Kubernetes 1.0 on CoreOS, we have put together some easy-to-read open source guides to run Kubernetes 1.0 on CoreOS. And as always if you need help try us on the #coreos irc channel or coreos-user mailing list.

CoreOS Joins Cloud Native Foundation

When we started building CoreOS Linux two years ago we wanted to encourage people to run infrastructure in a secure, distributed and consistent manner. This required many components along the way, including new datastores like etcd, container runtimes like Docker & rkt, and cluster wide application deployment, orchestration, and service discovery like Kubernetes. Today, CoreOS is joining a new foundation along with Google, Twitter, Huawei and other industry partners to collaborate and build the technologies that are changing how people are thinking about infrastructure software. This new foundation, the Cloud Native Foundation, is being launched in partnership with the Linux Foundation and will shepherd the industry collaboration around Kubernetes and other projects moving forward.

Tectonic Preview

For companies who want help building their infrastructure in this this manner we are also announcing that Tectonic is now in Preview, this includes: 24x7 support, a friendly web-based console, and deployment guides for AWS and on your own hardware. We invite you to read more about Tectonic Preview on our Tectonic blog.

Kubernetes Training

Also today, we are launching Kubernetes Training. The first workshops will be delivered by Kelsey Hightower, product manager, developer and chief advocate at CoreOS, and will take place on August 5 in Portland, August 7 in Seattle and August 10 in San Francisco.

By joining these workshops, you will learn more about Kubernetes, CoreOS, Docker and rkt and leave knowing Kubernetes Core Concepts, how to enable and manage key cluster add-ons such as DNS, monitoring, and the UI, how to configure nodes for the Kubernetes networking model and how to manage applications with Kubernetes deployment patterns.

For a limited time, the workshops will be available at a special rate for only the cost materials. Sign-up for a workshop in your area early they will fill-up fast.

CoreOS at OSCON

The CoreOS team is at OSCON this week and you have three ways to find us:

Attend Kelsey Hightower's tutorial titled "Taming microservices with CoreOS and Kubernetes" Tuesday at 1:30pm
Join us on Wednesday for our free CoreOS Portland OSCON meetup starting at 6:00pm at the Ecotrust Building
Come by the Tectonic by CoreOS booth (#900) to meet our team and ask questions, and celebrate the launch of Kubernetes 1.0

Meet CoreOS at OSCON and more

2015-07-17T00:00:00+00:00

Next week we are heading to Portland, Oregon for OSCON. We look forward to meeting fellow OSCON attendees and Portland friends at one of the below events, or at our booth (# 900) on the OSCON show floor, July 21-24. If you have questions about Kubernetes, CoreOS, Docker or rkt sign up for office hours at our booth and get one-on-one time with our team. Read on to see more about where we will be next week. See you then!

Sunday, July 19

Get revved up for OSCON and see Kelsey Hightower, product manager, developer and chief advocate at CoreOS, speak in a lightning talk at the NGINX Summit at 3 p.m. PT. Register here for your ticket.

Tuesday, July 21

CoreOS will be at the Kubernetes 1.0 event – be sure to get there in time for the keynote at 11 a.m. PT. Get your ticket before it sells out! If you can’t make it in person you can register for the live-stream. We’ll be there throughout the day and if you miss us at the event, connect with our team at the Kubernetes After Hours Party on Tuesday too.

At OSCON, Kelsey Hightower will deliver a much-requested 3.5-hour tutorial starting at 1:30 p.m. PT on taming microservices with CoreOS and Kubernetes.

The OSCON Expo Hours begin at 5 p.m. so meet us at our booth if you’re there early for the reception.

Wednesday, July 22 - Thursday, July 24

Our CoreOS booth will have expert engineers to answer your questions and get you started with Tectonic. Sign up for office hours and talk with a CoreOS expert to get all your Kubernetes, CoreOS, Docker and rkt questions answered. Visit us at booth 900 all day on Wednesday and Thursday and tweet to us @CoreOSLinux or @TectonicStack.

Wednesday, July 22

Join us for our second annual CoreOS Portland OSCON meetup starting at 6 p.m. PT at the Ecotrust Building. Brian “Redbeard” Harrington, principal architect at CoreOS, Brandon Philips, CTO of CoreOS, Kelsey Hightower, product manager at CoreOS, and Matthew Garrett, principal security engineer at CoreOS, will lead the talks of the evening. We thank our sponsors, Redapt and Couchbase for making the event possible and providing drinks and bites on the Ecotrust rooftop! RSVP here.

Thursday, July 23

After your day at OSCON, join us for a Birds of a Feather (BoF) session at 7 p.m. PT by Brian “Redbeard” Harrington, principal architect at CoreOS. He will have a lively interactive conversation with attendees and cover how to get started with CoreOS, CoreOS components and CoreOS best practices you want to learn about most.

Friday, July 24

At 11:10 a.m. PT Matthew Garrett will present building a trustworthy computer.

See you in Portland!

Announcing rkt v0.7.0, featuring a new build system, SELinux and more

2015-07-15T00:00:00+00:00

Today we are announcing rkt v0.7.0. rkt is an app container runtime built to be efficient, secure and composable for production environments. This release includes new subcommands for a rkt image to manipulate images from the local store, a new build system based on autotools and integration with SELinux. These new capabilities improve the user experience, make it easier to build future features and improve security isolation between containers.

Note on rkt and OCP

As you know, rkt is an implementation of the App Container (appc) spec and rkt is also targeted to be a future implementation of the Open Container Project (OCP) specification. The OCP development is still in its early days. Our plans with rkt are unchanged and the team is committed to the continued development of rkt. This is all a part of the goal to build rkt as a leading container runtime focused on security and composability for the most demanding production requirements.

Now, read on for details on the new features.

New Subcommands for rkt image

In this release all of the subcommands dealing with images in the local store can be found inside rkt image. Apart from the already existing subcommands rkt image list, rkt image rm and rkt image cat-manifest, this release adds three more:

rkt image export

This subcommand exports an ACI from the local store. This comes in handy when you want to copy an image to another machine, file server and so-on.

$ rkt image export coreos.com/etcd etcd.aci
$ tar xvf etcd.aci

Note that this command does not perform any network I/O so the image must be in the local store beforehand. Also, the exported ACI file might be different from the original imported to the store because rkt image export always returns uncompressed ACIs.

rkt image extract

For debugging or inspection you may want to extract an ACI to a directory on disk. You can get the full ACI or just its rootfs:

$ rkt image extract coreos.com/etcd etcd-extracted
$ find etcd-extracted
etcd-extracted
etcd-extracted/manifest
etcd-extracted/rootfs
etcd-extracted/rootfs/etcd
etcd-extracted/rootfs/etcdctl
...
$ rkt image extract --rootfs-only coreos.com/etcd etcd-rootfs
$ find etcd-rootfs
etcd-extracted
etcd-extracted/etcd
etcd-extracted/etcdctl
...

As with rkt image export no network I/O will be performed.

rkt image render

While the previous command extracts an ACI to a directory, it doesn’t take into account image dependencies or pathWhitelists. To get an image rendered as it would look ready-to-run inside of the rkt stage2 you can run rkt image render:

$ rkt image render --rootfs-only coreos.com/etcd etcd-rendered
$ find etcd-rendered
etcd-extracted
etcd-extracted/etcd
etcd-extracted/etcdctl
...

New Build System

In 0.7.0 we introduce a new build system based on autotools. Previous versions of rkt were built with a combination of shell scripts and ad-hoc Makefiles. As building complexity grew, more and more environment variables were added that made new build options less discoverable and complicated development.

The new build system based on autotools in 0.7.0 has more discoverable options and should make it easier to build future features like cross-compiling or a KVM-based stage1.

This is how you build rkt now:

$ ./autogen.sh 

----------------------------------------------------------------
Initialized build system. For a common configuration please run:
----------------------------------------------------------------

./configure --with-stage1=coreos
$ ./configure --help
`configure' configures rkt 0.7.0+git to adapt to many kinds of systems.
[...]
Optional Features:                                                                                                                                                                                                                                                             
  --disable-option-checking  ignore unrecognized --enable/--with options                                                                                                                                                                                                       
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)                                                                                                                                                                                                 
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]                                                                                                                                                                                                                            
  --enable-functional-tests                                                                                                                                                                                                                                                    
                          enable functional tests on make check (linux only,                                                                                                                                                                                                   
                          uses sudo, default: no, use auto to enable if                                                                                                                                                                                                        
                          possible)                                                                                                                                                                                                                                            

Optional Packages:                                                                                                                                                                                                                                                             
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-stage1=type      type of stage1 build one of 'src', 'coreos', 'host',
                          'none', 'kvm' (default: 'coreos')
  --with-stage1-systemd-src=git-path
                          address to git repository of systemd, used in 'src'
                          build mode (default: 'https://github.com/systemd/systemd.git')
  --with-stage1-systemd-version=version
                          systemd version to build (default:
                          'v220')
  --with-stage1-image-path
                          custom stage1 image path (default:
                          '')
$ ./configure && make -j4
[...]

Note that all the build options are listed with a description text that helps the user know what to write instead of having them read the build scripts to figure out which environmental variables to set.

SELinux Support

We also added support for running containers using SELinux SVirt, improving security isolation between containers. This means every rkt instance will run in a different SELinux context. Processes started in these contexts will be unable to interact with processes or files in any other instance’s context, even though they are running as the same user.

This feature depends on appropriate policy being provided by the underlying Linux distribution. If supported, a file called “lxc_contexts” will be present in the SELinux contexts directory under /etc/selinux. In the absence of appropriate support, SELinux SVirt will automatically be disabled at runtime.

Other Features

rkt registers pods with the metadata service by default now. Ensure it is running before running pods (rkt metadata-service) or disable registration with rkt run --mds-register=false.
We started improving rkt UX by reducing stage1 verbosity and writing better and more consistent error messages. As we look towards the next rkt releases, we will be focusing more on UX improvements.

Get Involved

Be a part of the development of rkt or join the discussion through the rkt-dev mailing list or GitHub issues. We welcome you to contribute directly to the project.

Q&A with Sysdig on containers, monitoring and CoreOS

2015-07-14T00:00:00+00:00

Today we congratulate Sysdig, the container visibility company, on its funding news and launch of its commercial offering, Sysdig Cloud. We interviewed Loris Degioanni, the creator and CEO of Sysdig, about the company, containers and how Sysdig works with CoreOS. He is a pioneer in the field of network analysis through his work on WinPcap and Wireshark, which are open source tools with millions of users worldwide.

Read on to dive in, and be sure to meet Sysdig and our team at our July 29 Meetup in San Francisco to learn more.

Q: In your own words, what is Sysdig? Why is it important in containerized environments?

Loris: Sysdig is an open source system visibility tool, designed to meet the needs of modern IT professionals. You can use it to monitor and troubleshoot things like system activity, network and file I/O, application requests and much more. Unique features include the ability to work with trace files (similar to tools such as Wireshark) and deep, native container support.

As for containerized environments: containers are an extremely interesting and powerful technology – I’m personally a big fan. But containers are also a relatively young technology (at least in their current form), and until now there has been a bit of catch 22 in terms of container visibility. Either you monitor your containers from the outside, with inherently limited visibility, given the opaque and self-contained nature of containers. Or you install extra monitoring software inside the container, which largely undermines the benefits of using a container in the first place – performance, deployability, portability, dependency simplification, security, etc.

Sysdig is the first visibility tool designed specifically to support containers. And in order to truly support containers, we knew we had to solve the issue above. Sysdig’s instrumentation is based on a little kernel module that can capture information like system calls from “underneath” containers. This makes it possible to explore anything that’s happening inside containers, while running sysdig entirely on the host machine or inside another container. There is no need to instrument your containers, or install any agent inside them. In other words, Sysdig provides full visibility into all your containers, from outside of the containers themselves.

This tends to be quite a radical departure from what people are used to, and is also the basis of our commercial product, Sysdig Cloud. Based on this same open source technology, Sysdig Cloud offers a container-native monitoring solution, with distributed collection, discovery, dashboarding, alerting, and topology mapping.

Q: What lessons from contributing to Wireshark influence what you are doing today?

Loris: I spent my Ph.D. and the first 10 years of my career working on network monitoring. The lessons I learned during that time have highly influenced the architecture and underlying philosophies of sysdig.

Network monitoring as a whole offers a pretty elegant set of workflows. First, there is the fundamental ability to capture the information you need into trace files. These trace files are not only easily shared, but maybe even more importantly, they decouple the troubleshooting process from the issue itself. No longer are you working inside of a broken system, trying to fix a problem, as the problem is bringing down the system around you. Network monitoring workflows also include the ability to filter information with well known languages, and visualize your data with industry standard tools like Wireshark.

I believe these workflows are not only relevant in the context of network monitoring. Trace files, decoupled troubleshooting, natural filters, standardized visualizations: these are widely applicable concepts. With our work on sysdig, we are trying to bring these well-proven approaches from the world of network monitoring into the world of system, container and application visibility.

Q: How does Sysdig work with CoreOS environments? What types of information can Sysdig pull from a CoreOS host?

Loris: Sysdig fully supports CoreOS environments, and offers the same 100% visibility you would find in a non-containerized environment. Sysdig works with CoreOS by installing the container we provide, which contains all the required dependencies and offers an isolated execution environment. Since we provide a precompile driver, installation is really easy – it is a single command line and takes 30 seconds.

Once installed, sysdig will be able to surface very rich information about your CoreOS environment: both the host OS and the containers you have running. This includes everything from top processes, to network connections, to top files and directories, to a list of executed commands for both the host OS and any of the running containers. And that’s just the tip of the iceberg. For some interesting use cases with sysdig running in CoreOS environments, you can refer to our two-part CoreOS blog series here and here.

Q: What is the memory and CPU overhead required by Sysdig?

Loris: Typically low, but it depends on what kind of activity is happening on the machine. Sysdig instruments the operating system’s kernel, and the overhead depends on how many events there are to be captured. On a machine with average load, the CPU occupation should be very low: a few percentage points. CPU occupation of sysdig can go higher on systems with a lot of I/O or network activity. The Sysdig Cloud agent, on the other hand, incorporates additional protective mechanisms, such as subsampling techniques, to ensure the CPU occupation always stays within an acceptable range of <5%.

Q: You presented the Dark Art of Container Monitoring at CoreOS Fest this year. Tell us more about what should be monitored.

Loris: In terms of what should be monitored, my answer is: everything! The really important question is: how should it be monitored? The same features that make containers so interesting and revolutionary (i.e. the fact that they are isolated, self-contained, simple and lightweight), make them a real challenge to monitor. In particular, the traditional approach of having an agent on any “entity” doesn’t work well with containers, because it’s too invasive and doesn’t scale.

This is the problem we’re trying to solve with sysdig and Sysdig Cloud. We’re excited about working on it because great visibility is a key requirement to adopt containers in production.

Q: Describe what Sysdig does with CoreOS Linux to help monitor system security.

Loris: Sysdig has powerful security-oriented features. Here are some examples of what CoreOS users can do with sysdig to monitor system security:

Show the directories that the user "root" visits
Observe ssh activity
Show every file open that happens in /etc
Show all the network connections established by a process

Now think about being able to obtain this information for any container running on a CoreOS host, but from outside the container, with no instrumentation and no dependencies.

If you are curious to try sysdig out, installation on CoreOS is super easy and instructions can be found here. And don’t forget to let us know what you think on twitter or at info@sysdig.com!

Join CoreOS and Sysdig in San Francisco for the July Meetup

Attend this month’s CoreOS San Francisco Meetup that will feature the CoreOS team and Gianluca Borello, senior software engineer at Sysdig.

When: Wednesday, July 29, 2015 starting at 6 p.m. PT

Where: Okta, 301 Brannan Street, San Francisco, CA 94107

RSVP: http://www.meetup.com/coreos/events/223897172/

How to get involved with CoreOS projects

2015-07-10T00:00:00+00:00

Today we’re excited to build and collaborate with our community at the inaugural CoreOS hackathon. Even if you can’t join us at GopherCon in Denver, there are numerous ways to get involved and contribute.

Every project on GitHub includes helpful information on contributing that can be found in the CONTRIBUTING.md file. Be sure to look at it before you jump in and begin coding.

etcd

Serving as the backbone of many distributed systems, from Kubernetes to Pivotal’s Cloud Foundry and beyond, etcd is a Go codebase and key-value store where the state of the art in distributed systems comes together. If your interests lie in consensus protocols, APIs and clustering, etcd has a number of areas where contribution is more than welcome.

Hack on etcd.

fleet

CoreOS Linux is built for scale, and at scale, managing systemd can be a challenge. We created fleet to distribute init across the data center. fleet is used in nearly all CoreOS deployments to simplify administrative overhead. If you are interested in operational plumbing, there is no shortage of work to be done.

Hack on fleet.

flannel

If software defined networks and the low levels of data center connectivity are in your interests, you can help build flannel, the container-friendly software networking fabric.

Hack on flannel.

rkt

Jump in with the rkt team and help create a secure, composable and standards based container runtime starting with these requests. Help shape the future of the container ecosystem and stay up to date with our rkt mailing list too.

With the recent announcement to collaborate on the Open Container Project (OCP), stay tuned for updates on how we will work together and work with OCP and rkt.

GopherCon Hack Day on July 10

For any of you attending GopherCon, here are the details of the hack day:

When: Friday, July 10, 2015 from 10:00 a.m. - 5:00 p.m. MDT

Where: Room 403, Denver Convention Center

Schedule:

10:00 a.m. - 10:30 a.m. - Brandon Philips, CoreOS Two Years in

10:30 a.m. - 11:00 a.m. - Kelsey Hightower, Kubernetes talk

11 a.m. - 11:30 a.m. - Russell Haering, ScaleFT

11:30 a.m. - 12 p.m. - Micha Leuffen, Wercker

-LUNCH-

1:00 p.m. - 4:00 p.m. - Hack Day Competition

4:00 p.m. - 5:00 p.m. - Competition demos and winner announcement

We welcome your involvement and contributions to CoreOS projects. We wouldn’t be here without our contributors and there is much to be done!

OpenSSL has been Updated (CVE-2015-1793)

2015-07-10T00:00:00+00:00

The Alternative Chains Certificate Forgery vulnerability in OpenSSL, as reported in CVE-2015-1793, has been patched in CoreOS Linux (Alpha, Beta and Stable channels). If automatic updates are enabled (default configuration), your server should be patched within the next several hours (if it hasn’t already received the update).

If automatic updates are disabled, you can force an update by running update_engine_client -check_for_update.

If you have any questions or concerns, please join us in IRC freenode/#coreos.

Happy 2nd Epoch CoreOS Linux

2015-07-07T00:00:00+00:00

Today we rolled out the 735.0.0 release of CoreOS Linux to the alpha channel. Our CoreOS Linux version numbers are counted from our epoch on July 1, 2013, which means this month marks the end of our second year working on CoreOS Linux.

Two years ago we started this journey with a vision of improving the consistency, deployment speed and security of server infrastructure. In this time we have kicked off a rethinking of how server OSes are designed and used. In a recent article InfoWorld said:

CoreOS Linux “was the first influential micro operating system designed for today’s cloud environments.”

Last year, we celebrated our first stable channel release and since then we have been hard at work pushing important bug fixes and feature releases to that channel every 2.5 weeks on average.

CoreOS Year 1 Highlights

In the post for that first stable release we highlighted our progress to date:

CoreOS engineers contributed features and fixes to open source projects including Docker, the Linux kernel, networkd, systemd and more
Official CoreOS image added to Google Compute Engine, Rackspace, Amazon
Joined the Docker Governance Board as a Contributing Member
Today’s most respected technology companies and many Fortune 500 companies are using and testing CoreOS in their environments

CoreOS Year 2 Highlights

In the tradition of that post one year ago, let’s take a look at some of the highlights from the last year of CoreOS.

Announced Tectonic, a commercial Kubernetes platform that combines the CoreOS stack with Kubernetes to bring companies Google-style infrastructure
Worked with community partners to create App Container (appc), a specification defining a container image format, runtime environment and discovery protocol, to work towards the goal of a standard, portable shipping container for applications
Created rkt, a container runtime designed for composability, security and speed and the first implementation of appc
Quay.io joined CoreOS to provide Enterprise Registry, delivering secured hosting of your container repositories behind the firewall
Released etcd 2.0, which powers important projects in the container and distributed systems ecosystem including the flannel, locksmith, fleet and Kubernetes projects. etcd also supports community projects like HashiCorp’s Vault, and Docker 1.7s networking backend.
Joined forces with industry leaders to launch the Open Container Project, chartered to establish common standards for software containers

Our ability to build and ship innovative and high-quality projects is due in large part to the feedback and interest from our community. Thank you for all of your help in contributing, bug testing, promoting and learning more about what we are doing here at CoreOS.

Celebrate With Us at GopherCon

We will be celebrating our second birthday with our friends at GopherCon in Denver. Swing by our booth to get a limited edition CoreOS GopherCon sticker. Or, join us at our birthday party, brought to you by our friends from Couchbase and Iron.io on Thursday, July 9, at 8 p.m. MDT. Lastly, don’t miss our hack day on Friday, July 10, where you can work alongside a CoreOS engineer, learn about our open source projects and compete for prizes.

RSVP for our Second Birthday Party

Thursday, July 9 at 8 - 11 p.m. MDT

Pizza Republica in Denver, Colorado. Sponsored by Couchbase and Iron.io.

http://www.eventbrite.com/e/coreos-2nd-birthday-at-gophercon-tickets-17419178231

CoreOS Birthday Hack Day

Friday, July 10 at 10 a.m. - 5 p.m. MDT

Room 403, GopherCon in Denver, Colorado

http://www.gophercon.com/

Upcoming CoreOS Events in July

2015-07-06T00:00:00+00:00

Need your CoreOS fix? Check out where you can find us this month!

Tuesday, July 7-Friday, July 10, 2015 - Denver, CO

We’re going to GopherCon! Be sure to stop by our booth to pick up some swag and say hello.

Tuesday, July 7, 2015 at 6:00 p.m. MDT - Denver, CO

Start your GopherCon experience the right way. Join Brian “Redbeard” Harrington and other awesome speakers at the GopherCon Kick off party!

Thursday, July 9, 2015 at 1:40 p.m. MDT - Denver, CO

Be sure to check out Barak Michener give a talk at GopherCon about Cayley and building a graph database.

Thursday, July 9, 2015 at 4:00 p.m. MDT - Denver, CO

Don’t miss Kelsey Hightower at GopherCon talking about betting the company on go and winning!

Thursday, July 9, 2015 at 8:00 p.m. MDT - Denver, CO

If you’re attending GopherCon, come celebrate our second birthday with us and our friends from Couchbase and Iron.io at Pizza Republica! Pizza, beer and video games included. RSVP here!

Friday, July 10, 2015 - 11:00 a.m. BRT - Porto Alegre, Brazil

Meet Matthew Garrett and discuss Free Software communities at FISL in Brazil. He’ll present, Using DRM technologies to protect users.

Friday, July 10, 2015 at 10:00 a.m. MDT - Denver, CO

End GopherCon with a good time! Join us at our Hack Day in room 403. We’ll have speakers from CoreOS and the community, as well as a special Hack Day competition.

Tuesday, July 14, 2015 at 6:00 p.m. IDT - Tel Aviv, Israel

If you’re in Tel Aviv, swing by the Docker Tel Aviv Meetup to hear Joey Schorr discuss the Quay.io container lifecylce.

Tuesday, July 14, 2015 at 5:00 p.m. EDT - Online

DataStax is hosting a webinar on leveraging Docker and CoreOS to provide always available Cassandra at Instaclustr. Register here!

Tuesday, July 21, 2015 at 10:00 a.m. PDT - Portland, OR

Join us as we celebrate Kubernetes 1.0! Come by in person at the event or after party if you’re at OSCON. If you can’t make it to Portland, not to worry. Register to watch the keynote here.

Tuesday, July 21, 2015 at 1:30 p.m. PDT - Portland, OR

Kelsey Hightower will be at OSCON giving a workshop on taming microservices with CoreOS and Kubernetes. Don’t miss it!

Thursday, July 23, 2015 at 7:00 p.m. BST - London, UK

We’re ending the month with our friends at the CoreOS London Meetup! Come hang out and learn more about Tectonic and how it combines Kubernetes and the CoreOS software portfolio.

Want more CoreOS in your city? Let us know! email us at community@coreos.com.

Introducing flannel 0.5.0 with AWS and GCE

2015-06-30T00:00:00+00:00

Last week we released flannel v0.5, a virtual network that gives a range of IP addresses to each host to use with container runtimes. We have been working hard to add features to flannel to enable a wider variety of use cases, such as taking advantage of cloud providers' networking capabilities, as part of the goal to enable containers to effectively communicate across networks and ensure they are easily portable across cloud providers.

With this in mind, flannel v0.5 includes the following new features:

support for Google Compute Engine (GCE),
a client/server mode and,
a multi-network mode.

Please refer to the readme for details on the client/server and the multi-network modes.

Try Out the New Release

In this post we will provide an overview of how to setup flannel on Amazon Virtual Private Cloud (Amazon VPC) backend introduced in flannel v0.4 and the newly added GCE backend.

When flannel runs the gce or the aws-vpc backend it does not create a separate interface as it does when running the udp or the vxlan backends.

This is because with gce and aws-vpc backends, there is no overlay or encapsulation and flannel simply manipulates the IP routes to achieve maximum performance.

Let’s get started with setting up flannel on GCE instances.

GCE Backend

From the Developers Console, we start by creating a new network.

Configure the network name and address range. Then add firewall rules to allow etcd traffic (tcp/2379), SSH, and ICMP. That's it for the network configuration. Now it’s time to create an instance. Let's call it demo-instance-1. Under the "Management, disk, networking, access & security options" make the following changes:

Select the "Network" to be our newly created network
Enable IP forwarding
Under "Access and Security" set the compute permissions to "Read Write" and remember to add your SSH key

Booting a new GCE instance

Security settings for a new instance

With the permissions set, we can launch the instance!

The only remaining steps now are to start etcd, publish the network configuration and lastly, run the flannel daemon. SSH into demo-instance-1 and execute the following steps:

Start etcd:

$ etcd2 -advertise-client-urls http://$INTERNAL_IP:2379 -listen-client-urls http://0.0.0.0:2379

Publish configuration in etcd (ensure that the network range does not overlap with the one configured for the GCE network)

$ etcdctl set /coreos.com/network/config '{"Network":"10.40.0.0/16", "Backend": {"Type": "gce"}}'

Fetch the 0.5 release using wget from here
Run flannel daemon:

$ sudo ./flanneld --etcd-endpoints=http://127.0.0.1:2379

Now make a clone of demo-instance-1 and SSH into it to run the these steps:

Fetch the 0.5 release as before.
Run flannel with the --etcd-endpoints flag set to the internal IP of the instance running etcd

Check that the subnet lease acquired by each of the hosts has been added!

GCE Routes

It’s important to note that GCE currently limits the number of routes per project to 100.

Amazon VPC Backend

In order to run flannel on AWS we need to first create an Amazon VPC. Amazon VPC enables us to launch EC2 instances into a virtual network, which we can configure via its route table.

From the VPC dashboard start out by running the "VPC Wizard":

Select "VPC with a Single Public Subnet"
Configure the network and the subnet address ranges

Creating a new Amazon VPC

Now that we have set up our VPC and subnet, let’s create an Identity and Access Management (IAM) role to grant the required permissions to our EC2 instances.

From the console, select Services -> Administration & Security -> IAM.

We first need to create a policy that we will later assign to an IAM role. Under "Create Policy" select the "Create Your Own Policy" option. The following permissions are required as shown below in the sample policy document.

ec2:CreateRoute
ec2:DeleteRoute
ec2:ReplaceRoute
ec2:DescribeRouteTables
ec2:DescribeInstances

{
    "Version": "2012-10-17",
    "Statement": [
    {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateRoute",
                "ec2:DeleteRoute",
                "ec2:ReplaceRoute"
            ],
            "Resource": [
                "*"
            ]
    },
    {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeRouteTables",
                "ec2:DescribeInstances"
            ],
            "Resource": "*"
    }
    ]
}

Note that although the first three permissions can be tied to the route table resource of our subnet, the ec2:Describe* permissions can not be limited to a particular resource. For simplicity, we leave the "Resource" as wildcard in both.

With the policy added, let's attach it to a new IAM role by clicking the "Create New Role" button and setting the following options:

Role Name: demo-role
Role Type: "Amazon EC2"
Attach the policy we created earlier

We are now all set to launch an EC2 instance. In the launch wizard, choose the CoreOS-stable-681.2.0 image and under "Configure Instance Details" perform the following steps:

Change the "Network" to the VPC we just created
Enable "Auto-assign Public IP"
Select IAM demo-role

Configuring AWS EC2 instance details

Under the "Configure Security Group" tab add the rules to allow etcd traffic (tcp/2379), SSH and ICMP.

Go ahead and launch the instance!

Since our instance will be sending and receiving traffic for IPs other than the one assigned by our subnet, we need to disable source/destination checks.

Disable AWS Source/Dest Check

All that’s left now is to start etcd, publish the network configuration and run the flannel daemon. First, SSH into demo-instance-1:

Start etcd:

$ etcd2 -advertise-client-urls http://$INTERNAL_IP:2379 -listen-client-urls http://0.0.0.0:2379

Publish configuration in etcd (ensure that the network range does not overlap with the one configured for the VPC)

$ etcdctl set /coreos.com/network/config '{"Network":"10.20.0.0/16", "Backend": {"Type": "aws-vpc"}}'

Fetch the latest release using wget from here
Run flannel daemon:

sudo ./flanneld --etcd-endpoints=http://127.0.0.1:2379

Next, create and connect to a clone of demo-instance-1. Run flannel with the --etcd-endpoints flag set to the internal IP of the instance running etcd.

Confirm that the subnet route table has entries for the lease acquired by each of the subnets.

AWS Routes

Keep in mind that the Amazon VPC limits the number of entries per route table to 50.

Note that these are just sample configurations, so feel free to try it out and set up what works best for you!

App Container and the Open Container Project

2015-06-22T00:00:00+00:00

Today we’re pleased to announce that CoreOS, Docker, and a large group of industry leaders are working together on a standard container format through the formation of the Open Container Project (OCP). OCP is housed under the Linux Foundation, and is chartered to establish common standards for software containers. This announcement means we are starting to see the concepts behind the App Container spec and Docker converge. This is a win for both users of containers and our industry at large.

In December 2014 we announced rkt, a new container runtime intended to address issues around security and composability in the container ecosystem. At the same time, we started App Container (appc), a specification defining a container image format, runtime environment and discovery protocol, to work towards the goal of a standard, portable shipping container for applications. We believe strongly that open standards are key to the success of the container ecosystem.

We created App Container to kickstart a movement toward a shared industry standard. With the announcement of the Open Container Project, Docker is showing the world that they are similarly committed to open standards. Today Docker is the de facto image format for containers, and therefore is a good place to start from in working towards a standard. We look forward to working with Docker, Google, Red Hat and many others in this effort to bring together the best ideas across the industry.

As we participate in OCP, our primary goals are as follows:

Users should be able to package their application once and have it work with any container runtime (like Docker, rkt, Kurma, or Jetpack)
The standard should fulfill the requirements of the most rigorous security and production environments
The standard should be vendor neutral and developed in the open

App Container

We believe most of the core concepts from App Container will form an important part of OCP. Our experience developing App Container will play a critical role as we begin collaboration on the OCP specification. We anticipate that much of App Container will be directly integrated into the OCP specification, with tweaks being made to provide greater compatibility with the existing Docker ecosystem. The end goal is to converge on a single unified specification of a standard container format, and the success of OCP will mean the major goals of App Container are satisfied. Existing appc maintainers Brandon Philips and Vincent Batts will be two of the initial maintainers of OCP and will work to harmonize the needs of both communities in the spirit of a combined standard. At the same time we will work hard to ensure that users of appc will have a smooth migration to the new standard.

Continuing work on rkt

CoreOS remains committed to the rkt project and will continue to invest in its development. Today rkt is a leading implementation of appc, and we plan on it becoming a leading implementation of OCP. Open standards only work if there are multiple implementations of the specification, and we will develop rkt into a leading container runtime around the new shared container format. Our goals for rkt are unchanged: a focus on security and composability for the most demanding production environments.

We are excited the industry is converging a format that combines the best ideas from appc, rkt and Docker to achieve what we all need to succeed: a well-defined shared standard for containers.

For more information and to see the draft charter and founding formation of the OCP, go to www.opencontainers.org.

Technology Preview: CoreOS Linux and xhyve

2015-06-11T00:00:00+00:00

Yesterday a new lightweight hypervisor for OS X was released called xhyve; if you are familiar with qemu-kvm on Linux, it provides a roughly similar experience. In this post we are going to show how to run CoreOS Linux under xhyve. While this is all very early and potentially buggy tech, we want to give you some tips on how to try CoreOS Linux with xhyve and run Docker or rkt on top.

xyhve is a port of bhyve, the FreeBSD hypervisor, to OS X. It is designed to run off-the-shelf Linux distros. We’ve made it possible to run it on CoreOS Linux so you can get the benefits of a lightweight Linux OS running under a lightweight hypervisor on Macs. It is now possible to launch a full local development or testing environment with just a few shell commands.

A few ideas we are thinking about:

Single command to launch CoreOS Linux images.
Easily launch a Kubernetes cluster right on your laptop.
An OS X native version of rkt that can run Linux applications inside xhyve.

Keep in mind that xhyve is a very new project, so much work still needs to be done. You must be running OS X Yosemite for this to work. Check out this page for step-by-step instructions on how to try it out.

A Quick Example

Currently, you need to build xhyve yourself:

$ git clone https://github.com/mist64/xhyve
$ cd xhyve
$ make
$ sudo cp build/xhyve /usr/local/bin/

Now we can install the initial CoreOS tooling for xhyve:

$ git clone https://github.com/coreos/coreos-xhyve.git
$ cd coreos-xhyve
$ ./coreos-xhyve-fetch
$ sudo ./coreos-xhyve-run

Type ip a in the console of the virtual machine to get its IP address.

Let’s run a simple Docker container:

$ docker -H<ip-of-virtual-machine>:2375 run -it --rm busybox

Please open issues with ideas for enhancements or use cases. We welcome contributions to the code, so please open a pull request if you have code to share.

etcd2 in the CoreOS Linux Stable channel

2015-06-09T00:00:00+00:00

This week marks a particularly special milestone for etcd2. Beginning today, etcd2 will be available in the CoreOS Linux Stable channel. This means that everyone will now be able to take advantage of etcd2, which we launched earlier this year.

etcd is an open source, distributed, consistent key-value store. It is a core component of CoreOS software that helps to facilitate safe automatic updates, coordinate work between hosts, and manage overlay networking for containers. To recap, new features and improvements in etcd2 include:

Reconfiguration protocol improvements, enabling more safeguards against accidental misconfiguration
A new raft implementation, providing improved cluster stability and predictability in massive server environments
On-disk safety improvements, in which CRC checksums and append-only log behavior allow etcd to detect external data corruption and avoid internal file misoperations

More details can be found in this post which first introduced etcd2. Give it a shot and let us know what you think!

A special thank-you to all of the contributors who made this possible. Join us in the continued development of etcd through the etcd-dev discussion mailing list, GitHub issues, or contributing directly to the project.

Building and deploying minimal containers on Kubernetes with Quay.io and wercker

2015-06-03T00:00:00+00:00

Today's guest post has been written by Micha "mies" Hernandez van Leuffen, the founder and CEO of wercker, a platform and tool for building, testing and deploying in the modern world of microservices, containers and clouds.

Edit: Added video to end of post. Skip to video

The landscape of production has changed: monolithic is out, loosely coupled microservices are in. Modern applications consist of multiple moving parts, but most of the existing developer tooling we use was designed and built in the world of monolithic applications.

Working with microservices poses new challenges: your applications now consist of multiple processes, multiple configurations, multiple environments and more than one codebase.

Containers offer a way to isolate and package your applications along with their dependencies. Docker and rkt are popular container runtimes and allow for a simplified deployment model for your microservices. Wercker is a platform and command line tool built on Docker that enables developers to develop, test, build and deliver their applications in a containerized world. Each build artifact from a pipeline is a container, which gives you an immutable testable object linked to a commit.

In this tutorial, we will build and launch a containerized application on top of Kubernetes. Kubernetes is a cluster orchestration framework started by Google, specifically aimed at running container workloads. We will use quay.io from CoreOS for our container registry and wercker (of course!) to build the container and trigger deploys to Kubernetes.

The workflow we will create is depicted below:

Workflow from build to deployment.

Requirements

This tutorial assumes you have the following set up:

A wercker account. You can sign up for free here.
An account on quay.io.
A Kubernetes cluster. See the getting started section to set one up.
A fork of the application we will be building which you can find on GitHub.
You've added the above application to wercker and are using the Docker stack to build it.

Getting started

The application we will be developing is a simple API with one endpoint, which returns an array of cities in JSON. You can check out the source code for the API on GitHub. The web process listens on port 5000; we'll need this information later on.

Now, let's create our Kubernetes service configuration and include it into our repository.

{
   "kind": "Service",
   "apiVersion": "v1beta3",
   "metadata": {
      "name": "cities",
      "labels": {
         "name": "cities"
      }
   },
   "spec":{
      "createExternalLoadBalancer": true,
      "ports": [
         {
           "port": 5000,
           "targetPort": "http-server",
           "protocol": "TCP"
         }
      ],
      "selector":{
         "name":"cities"
      }
   }
}

We define the port that our application is listening on and use the public IP addresses that we got upon creating our Kubernetes cluster. We're using Google Container Engine, which allows for createExternalLoadBalancer. If you're using a platform which doesn't support createExternalLoadBalancer then you need to add the public IP addresses of the nodes to the publicIPs property.

Next, we're going to define our pipeline, which describes how wercker will build and deploy your application.

wercker.yml - build pipeline

On wercker, you structure your pipelines in a file called wercker.yml. It’s where you define the actions (steps) and environment for your tasks (tests, builds, deploys). Pipelines can either pass or fail, depending on the results of the steps within. Steps come in three varieties; steps from the wercker step registry, inline script steps and internal steps that run with extra privileges.

Pipelines also come with environment variables, some of which are set by default, others you can define yourself. Each pipeline can have its own base container (the main language environment of your application) and any number of services (databases, queues).

Now, let's have a look at our build pipeline for the application. You can check out the entire wercker.yml on GitHub.

build:
    box: google/golang
    steps:

    # Test the project
    - script:
        name: go test
        code: go test ./...

    # Statically build the project
    - script:
        name: go build
        code: CGO_ENABLED=0 go build -a -ldflags '-s' -installsuffix cgo -o app .

    # Create cities-controller.json only for initialization
    - script:
        name: create cities-controller.json
        code: ./create_cities-controller.json.sh

    # Copy binary to a location that gets passed along to the deploy pipeline
    - script:
        name: copy binary
        code: cp app cities-service.json cities-controller.json "$WERCKER_OUTPUT_DIR"

The box is the container and environment in which the build runs. Here we see that we're using the google/golang image as a base container for our build as it has the golang language and build tools installed in it. We also have a small unit test inside of our code base which we run first. Next we compile our code and build the app executable.

As we want to build a minimal container, we will statically compile our application. We disable the ability to create Go packages that call C code with the CGO_ENABLED=0 flag, rebuild all dependencies with the -a flag, and finally we remove any debug information with the -ldflags flag, resulting in an even smaller binary.

Next, we create our Kubernetes replication controller programmatically based on the git commit using a shell script. You can check out the shell script on GitHub.

The last step copies the executable and Kubernetes service definitions into the $WERCKER_OUTPUT_DIR folder, and the contents of this folder gets passed along to the /pipeline/source/ folder within the deploy pipeline.

wercker.yml - push to quay.io

We're now ready to set up our deploy pipelines and targets. We will create two deploy targets. The first will push our container to Quay.io, the second will perform the rolling update to Kubernetes. Deploy targets are created in the wercker web interface and reference the corresponding section in the wercker.yml.

Deploy targets in werker.

In order to add any information such as usernames, passwords, or tokens that our deploy target might need, we define these as environment variables for each target. These environment variables will be injected when a pipeline is executed.

Quay.io is a public and private registry for Docker image repositories. We will be using Quay.io to host the container image that is built from wercker.

deploy:
    box: google/golang
    steps:
     # Use the scratch step to build a container from scratch based on the files present
    - internal/docker-scratch-push:
        username: $QUAY_USERNAME
        password: $QUAY_PASSWORD
        cmd: ./app
        tag: $WERCKER_GIT_COMMIT
        ports: "5000"
        repository: quay.io/wercker/wercker-kubernetes-quay
        registry: https://quay.io

The deploy section of our wercker.yml above consists of a single step. We use the internal/docker-scratch-push step to create a minimal container based on the files present in the $WERCKER_ROOT environment variable (which contains our binary and source code) from the build, and push it to Quay.io. The $QUAY_USERNAME and $QUAY_PASSWORD parameters are environment variables that we have entered on the wercker web interface. For the tag, we use the git commit hash, so each container is versioned. This hash is available as an environment variable from within the wercker pipeline.

The cmd parameter is the command that we want to run on start-up of the container, which in our case is our application that we've built. We also need to define the port on which our application will be available, which should be the same port as in our Kubernetes service definition. Finally, we fill in the details of our Quay.io repository and the URL of the registry.

If you take a look at your Quay.io dashboard you will see that the final container that was pushed is just 1.2MB!

wercker.yml - Kubernetes rolling update

For this tutorial, we assume you've already created a service with an accompanying replication controller. If not, you can do this via wercker as well. See the initialize section in the wercker.yml

Let's proceed to do the rolling update on Kubernetes, replacing our pods one-by-one.

   rolling-update:
    - kubectl:
        server: $KUBERNETES_MASTER
        username: $KUBERNETES_USERNAME
        password: $KUBERNETES_PASSWORD
        insecure-skip-tls-verify: true
        command: rolling-update cities
        image: quay.io/wercker/wercker-kubernetes-quay:$WERCKER_GIT_COMMIT

The environment variables are again defined in the wercker web interface. The $KUBERNETES_MASTER environment variable contains the IP address of our instance.

Kubernetes credentials defined in the pipeline.

We execute the rolling update command and tell Kubernetes to use our Docker container from Quay.io with the image parameter. The tag we use for the container is the git commit hash.

Conclusion

In this tutorial, we have showcased how to build minimal containers and use wercker as our assembly line. Our final container was just 1.2MB, making for low-cost deploys!

Though the go programming language compiles to single binaries, making our life easier, our learnings can be applied to other programming languages as well.

Using wercker's automated build process we've not only created a minimal container, but also linked our artifact versioning to git commits in Quay.io.

Pairing our versioned containers with Kubernetes' orchestration capabilities results in a radically simplified deployment process, especially with the power of rolling updates!

In short, the combination of Kubernetes, Quay.io and wercker is a powerful and disruptive way of building and deploying modern-day applications.

In this article we've just scratched the surface of developing container-based microservices. To learn more about Kubernetes check out the getting started guides. For more information on Quay.io, see the documentation site. You can sign up for wercker here and more information and documentation is available at our dev center. The source code for our final application including its wercker.yml is available on GitHub.

Oh, the places we’ll be in June

2015-06-02T00:00:00+00:00

We’re across the US and in the Netherlands this month. Check out where we’re speaking!

Couchbase Connect: Thursday, June 4 at 1:45 p.m. PDT – Santa Clara, CA

Brian Harrington (@brianredbeard), also known as Redbeard, principal architect at CoreOS, will be at Couchbase Connect and will join Traun Leyden from Couchbase to discuss Tectonic, provide a deep dive on the technology behind Kubernetes, and walk through the steps required to get Couchbase running on Kubernetes.

HP Discover: Thursday, June 4 at 3:30 p.m. PDT – Las Vegas, NV

At HP Discover in Las Vegas this week? Brandon Philips (@brandonphilips), CTO of CoreOS, Janne Heino of Nokia and Chris Grzegorczyk (@grze), chief architect at HP, will speak on Thursday, June 4 at 3:30 p.m. at Discover Theater 1 about Hybrid cloud and containers for modern application architectures. Join Nokia to walk through its global private cloud deployment of Helion Eucalyptus that also uses CoreOS’s container runtime, rkt.

ContainerDays Boston: Friday, June 5 at 3:40 p.m. EDT – Boston, MA

Barak Michener (@barakmich), software engineer and CoreOS developer advocate, will be at ContainerDays Boston and will discuss CoreOS: Building the Layers of the Cluster. Barak will also join Dave Nielsen (@davenielsen) from CloudCamp on Saturday at 12:55 p.m. EDT for a workshop that will help you get started with deploying your first container to CoreOS, Cloud Foundry, Azure and AWS.

QCon: Monday, June 8 at 9 a.m. EDT – New York, NY

Join Kelsey Hightower (@kelseyhightower), product manager, developer and chief advocate at CoreOS, for an in-depth, day-long tutorial at QCon New York on Kubernetes and CoreOS.

Cloud Expo East: Tuesday, June 9 at 1:55 p.m. EDT – New York, NY

Meet Jake Moshenko, product technical lead at CoreOS, who will speak at Cloud Expo East about Containers: New Ways to Deploy and Manage Applications at Scale.

Nutanix .NEXT: Tuesday, June 9-Wednesday, June 10 – Miami, FL

Kelsey Hightower (@kelseyhightower) and Alex Polvi (@polvi), CEO of CoreOS, will present at Nutanix .NEXT, the company’s first user conference. See Kelsey speak on Tuesday, June 9 at 3:30 p.m. EDT on Containers—What They Mean for the Future of Application Deployment. Alex will join Alan Cohen, chief commercial officer at Illumio, Dheeraj Pandey (@trailsfootmarks), CEO of Nutanix, and JR Rivers (@JRCumulus), CEO of Cumulus Networks, in the closing keynote panel: The New Enterprise IT Stack. Don’t miss it on Wednesday, June 10 at 12:15 p.m. EDT.

GoSV Meetup: Tuesday, June 9 at 6:30 p.m. PDT – San Mateo, CA

The CoreOS team will be talking with the Go Silicon Valley Meetup group this month in San Mateo at Collective Health. Register here.

NYLUG Meetup: Wednesday, June 17 at 6:30 p.m. EDT – New York, NY

The CoreOS New York team will be at the New York Linux Users Group (NYLUG) and will provide an overview of CoreOS. Sign-ups begin on June 3. Register to attend here.

GoSF Meetup: Wednesday, June 17 at 6:30 p.m. PDT – San Francisco, CA

See the CoreOS team at the GoSF Meetup to listen in on a talk about A Survey of RPC options in Go.

GOTO Amsterdam: Friday, June 19 – Amsterdam, The Netherlands

Kelsey Hightower (@kelseyhightower) will be at GOTO Amsterdam speaking on rkt and the App Container spec at 11:30 a.m. CEST and will join a panel at 3:50 p.m. CEST to discuss Docker predictions.

Pre-DockerCon panel: Sunday, June 21 – San Francisco, CA

Join Kelsey Hightower (@kelseyhightower) and other thought leaders that will be at DockerCon for a pre-event evening panel on conducting systems and services: an evening about orchestration.

DevOpsDays Amsterdam: Wednesday, June 24 – Amsterdam, The Netherlands

Learn about CoreOS at a DevOpsDays Amsterdam workshop presented by Chris Kühl (@blixtra) on June 24.

In case you missed it, check out the recordings of the CoreOS Fest talks that were held last month. More will be posted this month so stay tuned.

CoreOS Linux is in the OpenStack App Marketplace

2015-05-19T00:00:00+00:00

Today at the OpenStack Summit in Vancouver, we are pleased to announce that CoreOS Linux – the lightweight operating system that provides stable, reliable updates to all machines connected to the update service – is included in the OpenStack Community App Catalog.

CoreOS Linux is now available in the Community App Catalog alongside ActiveState Stackato, Apcera, Cloud Foundry, Kubernetes, MySQL, Oracle Database 12c and Oracle Multitenant, Postgres, Project Atomic, Rally, Redis, Tomcat and Wordpress. The Community App Catalog is where community members can share apps and tools designed to integrate with OpenStack Clouds.

With the ability to use CoreOS directly from the catalog, it will be easier to use CoreOS Linux on OpenStack. CoreOS Linux delivers automatic updates that are critical to keeping a system secure. CoreOS Linux’s continuous stream of updates minimizes the complexity of an update and engineering teams have the flexibility to select specific release channels to deploy and control how clusters apply updates. Get started with CoreOS on OpenStack here.

At the intersection of open source technologies, we are excited to continue helping users succeed with containers in the OpenStack ecosystem. If you are at the OpenStack Summit this week, stop by to meet us and see our talk today at 2 p.m., Dream Stack, CoreOS + OpenStack + Kubernetes.

CoreOS at OpenStack Summit 2015

2015-05-18T00:00:00+00:00

CoreOS is in Vancouver this week. Not only are we excited to see where OpenStack is taking containers; we’re also pumped for 24-hour poutine.

There are CoreOS-focused events on the first three days of the conference! Speakers on Monday and Tuesday, plus a deep dive into CoreOS all afternoon on Wednesday.

Monday, May 18, 2015 at 2:00 p.m.

Don’t miss our very own Matthew Garrett talking about how we can secure cloud infrastructure using TPMs today at 2 p.m.

Tuesday, May 19, 2015 at 2:00 p.m.

Next up on Tuesday, May 19 at 2 p.m., we have Brian “Redbeard” Harrington from CoreOS telling us all about the Dream Stack, CoreOS + OpenStack + Kubernetes.

Wednesday, May 20, 2015 at 1:50-6:00 p.m.

To dive deeper into CoreOS, our Collaboration Day event is on Wednesday at 1:50-6 p.m. Brian Harrington and Brian Waldon will be there to answer all of your CoreOS questions. Here is the schedule:

Time	Topic
1:50 - 2:30	CoreOS as a building block for OpenStack Ironic
2:40 - 3:20	Managing CoreOS Images effectively with Glance (Dos and Don'ts)
3:30 - 4:10	CoreOS Developer AMA (Ask Me Anything)
4:10 - 4:30	20 minute break
4:30 - 5:10	Administrative/Firmware containers - going beyond your web applications
5:20 - 6:00	Building minimal application containers from scratch

Be sure to stop by our 3D-printing booth right near registration! That’s right. We’ve teamed up with Codame to immortalize your time here at OpenStack Summit. Try it alone, or bring a friend. You’re not going to want to miss this!

Meet our team and tweet to us @CoreOSLinux!

New Functional Testing in etcd

2015-05-14T00:00:00+00:00

Today we are discussing the new fault-injecting, functional testing framework built to test etcd, which can deploy a cluster, inject failures, and check the cluster for correctness continuously.

For context, etcd is an open source, distributed, consistent key-value store. It is a core component of CoreOS software that facilitates safe automatic updates, coordinates work scheduled to hosts, and sets up overlay networking for containers. Because of its core position in the stack, its correctness and availability is significantly critical, which is why the etcd team has built the functional testing framework.

Since writing the framework, we have run it continuously for the last two months, and etcd has shown to be robust under many kinds of harsh failure scenarios. This framework has also helped us identify a few potential bugs and improvements that we’ve fixed in newer releases — read on for more info.

Functional Testing

etcd’s functional test suite tests the functionality of an etcd cluster with a focus on failure-resistance under heavy usage.

The main workflow of the functional test suite is straightforward:

It sets up a new etcd cluster and injects a failure into the cluster. A failure is some unexpected situation that may happen in the cluster, e.g., machine doesn’t work or network is down.
It repairs the failure and expects the etcd cluster to recover within a short amount of time (usually one minute).
It waits for the etcd cluster to be fully consistent and making progress.
It starts the next round of failure injection.

Meanwhile, the framework makes continuous write requests to the etcd cluster to simulate heavy workloads. As a result, there are constantly hundreds of write requests queued, intentionally causing a heavy burden on the etcd cluster.

If the running cluster cannot recover from failure, the functional testing framework archives the cluster state and does the next round of testing on a new etcd cluster. When archiving, process logs and data directories for each etcd member are saved into a separate directory, which can be viewed and debugged in the future.

Basic Architecture

etcd's functional test suite has two components: etcd-agent and etcd-tester. etcd-agent runs on every etcd node and etcd-tester is a single controller of the test.

etcd-agent is a daemon on each machine. It can start, stop, restart, isolate and terminate an etcd process. The agent exposes these functionalities via RPC.

etcd-tester utilizes all etcd-agents to control the cluster and simulate various test cases. For example, it starts a three-member cluster by sending three start-RPC calls to three different etcd-agents. It then forces one of the members to fail by sending a stop-RPC call to the member’s etcd-agent.

While etcd-tester uses etcd-agent to control etcd externally, it also directly connects to etcd members to make simulated HTTP requests, including setting a range of keys and checking member health.

Internal Testing Suite

The internal functional testing suite case is built upon four n1-highcpu-2 virtual machines on Google Compute Engine. Each machine has 2 virtual cores, 1.8G memory and 200G standard persistent disk. Three machines have etcd-agent running as a daemon, while the fourth machine runs etcd-tester as the controller.

Currently we have six major failures to simulate the most common cases that etcd may meet in real life:

kill all members
- the whole data center experiences an outage, and the etcd cluster in the data center is killed
kill the majority of the cluster
- part of the data center experiences an outage, and the etcd cluster loses quorum
kill one member
- a single machine needs to be upgraded or maintained
kill one member for a significant time and expect it to recover from an incoming snapshot
- a single machine is down due to hardware failure, and requires manual repair
isolate one member
- the network interface on a single machine is broken
isolate all members
- the router or switch in the data center is broken

Meanwhile, 250k 100-byte keys are written into the etcd cluster continuously, which means we’re storing about 25MB of data in the cluster.

Discovering Potential Bugs

This test suite has helped us to discover potential bugs and areas to improve. In one discovery, we found that when a leader is helping the follower catch up with the progress of the cluster, there was a slight possibility that memory and CPU usage could explode without bound. After digging into the log, it turned out that the leader was repeatedly sending 50MB-size snapshot messages and overloaded its transport module. To fix the issue, we designed a message flow control for snapshot messages that solved the resource explosion.

Another example is the automatic WAL repair feature added in 2.1.0. To protect data integrity, etcd intentionally refuses to restart if the last entry in the underlying WAL was half-written, which may happen if the process is killed or disk is full. We've found this happens occasionally (once per hundred rounds) in functional testing, and it’s safe and easier to remove the error automatically and recover from the cluster to simplify the recovery for the administrator. This functionality has been merged into the master branch, and will be released in v2.1.0.

After several weeks of running and debugging, the etcd cluster has survived several thousand consecutive rounds of all six failures. Surviving serious testing, the etcd cluster is strong and working quite well.

Diving into the Code

Build and Run

etcd-agent can be built via

$ go build github.com/coreos/etcd/tools/functional-tester/etcd-agent

and etcd-tester at

$ go build github.com/coreos/etcd/tools/functional-tester/etcd-tester

Run etcd-agent binary on machine{1,2,3}:

$ ./etcd-agent --etcd-path=$ETCD_BIN_PATH

Run etcd-tester binary on machine4:

$ ./etcd-tester -agent-endpoints=”$MACHINE1_IP:9027,$MACHINE2_IP:9027,$MACHINE3_IP:9027” -limit=3 -stress-key-count=250000 -stress-key-size=100

etcd-tester starts running, and makes 3 rounds of all failures on a 3-member cluster in machines 1, 2, and 3.

Add a new failure

Let us go through the process to add failureKillOne, which kills one member and recovers it afterwards. First, write how to inject and recover from failure:

type failureKillOne struct {
  description
}

func newFailureKillOne() *failureKillOne {
  return &failureKillOne{
    // detailed description of the failure
    description: "kill one member",
  }
}

func (f *failureKillOne) Inject(c *cluster, round int) error {
  // round robin on all members
  i := round % c.Size
  // ask its agent to stop etcd
  return c.Agents[i].Stop()
}

func (f *failureKillOne) Recover(c *cluster, round int) error {
  i := round % c.Size
  // ask its agent to restart etcd
  if _, err := c.Agents[i].Restart(); err != nil {
    return err
  }
  // wait for recovery done
  return c.WaitHealth()
}

Then we add it into failure lists:

  t := &tester{
    failures: []failure{
      newFailureKillOne(),
    },
    cluster: c,
    limit:   *limit,
  }

Done.

As you see, the framework is simple but already fairly powerful. We are looking forward to having you join the etcd test party!

Future Plans

The framework is still under active development, and more failure cases and checks will be added.

Random network partitions, network delays and runtime reconfigurations are some classic failure cases that the framework does not yet cover. Another interesting idea we plan to explore is a cascading failure case that injects multiple failure cases at the same time.

On the recovery side, more checks against consistent views of the keyspace on all members is a good starting point for more exploration.

The internal testing cluster runs 24/7, and our etcd cluster works perfectly under the current failure set. The etcd team is making its best effort to guarantee etcd’s correctness, and hopes that we can provide users the most robust consensus store possible.

Follow-up plans for more specific and harsher tests are in our TODO list. This framework is good to imitate real-life scenarios, but it cannot have fine controls on lower-level system and hardware behaviors. Future testing approaches may use simulated networks and disks to tackle these failure simulations.

We will keep enhancing the testing strength and coverage by adding more failure cases and checks into the framework. Pull requests to the framework are welcomed!

Acknowledgement

We are running our testing cluster on GCE. Thanks to Google for providing the testing environment.

Upcoming CoreOS Events in May

2015-05-12T00:00:00+00:00

We kicked off May by hosting our first ever CoreOS Fest, and it was a blast! We’re sad to see it go, but we’re excited about all of the other events we’ll be speaking at and attending this month.

Wednesday, May 13, 2015 at 2:00 p.m. EDT

What could be better than listening to Kelsey Hightower give a talk! Listen in from anywhere in the world to hear Kelsey discuss how to get started with containers and microservices during the Logentries Webinar. Register now!

Wednesday, May 13, 2015 at 6:00 p.m. PDT - San Francisco, CA

Alex Crawford from CoreOS will be giving an overview of CoreOS at the SF DevOps Meetup group. Thanks to Teespring for hosting the event at its SOMA office. Be sure not to miss it!

Tuesday, May 19, 2015 at 2:00 p.m. PDT - Vancouver, BC Canada

If you find yourself at OpenStack Summit Vancouver, be sure to check out Brian ‘Redbeard’ Harrington talk about modern practices for building a private cloud that runs containers at scale. We’ll also have our team there, so please stop by our area and meet us. We even have a Collaboration Day session for attendees on Wednesday, May 20 from 1:50 p.m. to 6 p.m.

Wednesday, May 20, 2015 at 9:10 a.m. EDT - Seven Springs, PA

CoreOS CEO Alex Polvi will be keynoting WHD.usa this year by talking about building distributed systems and securing the backend of the internet.

Wednesday, May 20, 2015 at 6:30 p.m. EDT - Atlanta, GA

Join Brian Akins from CoreOS in Atlanta at the DevOps ATL Meetup, where he’ll be discussing new ways to deploy and manage applications at scale. Thanks to MailChimp for hosting this meetup at their Ponce City Market office.

Wednesday, May 20, 2015 at 11:05 a.m. MDT - Denver, CO

Don’t miss Kelsey Hightower at GlueCon 2015 where he’ll give an overview of key technologies at CoreOS and how you can use these new technologies to build performant, reliable, large distributed systems.

Thursday, May 21 2015 at 11:20 a.m. MDT - Denver, CO

CoreOS CTO Brandon Philips will be at GlueCon 2015 discussing how to create a Google-like infrastructure. It will cover everything you need to know from the OS to the scheduler.

Thursday, May 21 2015 at 2:40 p.m. EDT - Charleston, SC

You can find Kelsey Hightower at CodeShow SE 2015 explaining how to manage containers at scale with CoreOS and Kubernetes.

Thursday, May 21, 2015 at 7:30 p.m. CEST - Madrid, Spain

Iago Lopez Galeiras will be joining the Madrid DevOps Meetup this month to give a talk on rkt and the App Container spec.

Tuesday, May 26, 2015 at 6:00 p.m. EDT - Charlottesville, VA

Don’t miss Brian Akins from CoreOS give an introduction to building large reliable systems at the DevOps Charlottesville Meetup group.

Friday, May 29, 2015 at 11:50 a.m. PDT - Santa Clara, CA

Be sure to check out Kelsey Hightower at Velocity where his talk will examine all the major components of CoreOS including etcd, fleet, docker, and systemd; and how these components work together.

CoreOS Fest Recap

Check out some of the best moments from CoreOS Fest 2015!

Join us at an event in your area! If you would like our help putting together a CoreOS meetup, or would like to speak at one of our upcoming meetups, please contact us at press@coreos.com.

CoreOS State of the Union at CoreOS Fest

2015-05-05T00:00:00+00:00

At CoreOS Fest we have much to celebrate with the open source community. Today over 800 people contribute to CoreOS projects and we want to thank all of you for being a part of our community.

We want to take this opportunity to reflect on where we started from with CoreOS Linux. Below, we go into depth about each project, but first, a few highlights:

We've now shipped CoreOS Linux images for nearly 674 days, since the CoreOS epoch on July 1, 2013.
We've rolled out 13 major releases of the Linux kernel from 3.8.0, released in February 2013, to the 4.0 release in April 2015.
In that time, we have tagged 329 releases of our images.
We have 500+ projects on GitHub that mention etcd, including major projects like Kubernetes, using etcd.

CoreOS Linux

Our namesake project, CoreOS Linux, started with the idea of continuous delivery of a Linux operating system. Best practice in the industry is to ship applications regularly to get the latest security fixes and newest features to users – we think an operating system can be shipped in a similar way. And for nearly two years, since the CoreOS epoch on July 1, 2013, we have been shipping regular updates to CoreOS Linux machines.

In a way, CoreOS Linux is a kernel delivery system. The alpha channel has rolled through 13 major releases of the Linux kernel from 3.8.0 in February 2013 to the recent 4.0 release in April 2015. This doesn’t include all of the minor patch releases we have bumped through as well. In that time we have tagged 329 releases of our images. To achieve this goal, CoreOS uses a transactional system so upgrades can happen automatically.

CoreOS Linux stats shared at CoreOS Fest

Community feedback has been incredibly important throughout this journey: users help us track down bugs in upstream projects like the Linux kernel, give us feedback on new features, and flag regressions that are missed by our testing.

A wide variety of companies are building their products and infrastructure on top of CoreOS Linux, including many participants at CoreOS Fest:

Deis, a project recently acquired by Engine Yard, spoke yesterday on "Lessons Learned From Building Platforms on Top of CoreOS" Mesosphere DCOS uses CoreOS by default, and we are happy to have them sponsor CoreOS Fest Salesforce Data.com spoke today on how they are using distributed systems and application containers Coinbase presented a talk today on "Container Management & Analytics"

etcd

We build CoreOS Linux with just a single-host use case in mind, but wanted people to trust and use CoreOS to update their entire fleet of machines. To solve this problem of automated yet controlled updates across a distributed set of systems, we built etcd.

etcd was initially created to provide an API-driven distributed "reboot lock" to a cluster of hosts, and it has been very successful serving this basic purpose. But over the last two years, adoption and usage of etcd has been exploded: today it is being used as a key part of projects like Google's Kubernetes, Cloud Foundry's Diego, Mailgun's Vulcan and many more custom service discovery and master election systems.

At CoreOS Fest we have seen demonstrations of a PostgreSQL master election system built by Compose.io, a MySQL master election system built by HP, and a discussion by Yodlr about how they use it for their internal microservice infrastructure. With feedback from all of these users of etcd, we are planning an advanced V3 API, a next-generation disk-backed store and writing new punishing long-running tests to ensure etcd remains a highly reliable component of distributed infrastructure.

etcd stats shared at CoreOS Fest

fleet on top of etcd

After etcd, we built fleet, a scheduler system that ties together systemd and etcd into a distributed init system. fleet can be thought of as a logical extension of systemd that operates at the cluster level instead of the machine level.

The fleet project is low level and designed as a foundation for higher order orchestration: its goal is to be a simple and resilient init system for your cluster. It can be used to run containers directly and also as a tool to bootstrap higher-level software like Kubernetes, Mesos, Deis and others.

For more on fleet, see the documentation on launching containers with fleet.

fleet stats shared at CoreOS Fest

rkt

The youngest CoreOS project is rkt, a container runtime, which was launched in December. rkt has security as a core focus and was designed to fit into the existing Unix process model to integrate well with tools like systemd and Kubernetes. And rkt was also built to support the concept of pods: a container composed of multiple processes that share resources like local network and IPC.

Where is rkt today? At CoreOS fest we discussed how rkt was integrated into Kubernetes, and showed this functionality in a demo yesterday. rkt is also used in Tectonic, our new integrated container platform. Looking forward, we are planning improved UX around trust and image handling tools, advanced networking capabilities, and splitting the stage1 out from rkt to support other isolation mechanisms like KVM.

rkt stats shared at CoreOS Fest

Container networking

Containers are most useful when they can interact with other systems over the network. Today in the container ecosystem we have some fairly basic patterns for network configuration, but over time we will need to give users the ability to configure more complex topologies. CNI (Container Network Interface) defines the API between a runtime like rkt and how a container actually joins a network, via an external plugin interface. Our intention with CNI is to develop a generic networking solution supporting a variety of tools, with reusable plugins for different backend technologies like macvlan, ipvlan, Open vSwitch and more.

flannel is another important and useful component in container network environments. In our future work with flannel, we’d like to introduce a flannel server, integrate it into Kubernetes and add generic UDP encapsulation support.

Ignition: Machine Configuration

Ignition is a new utility for configuring machines on first boot. This utility provides similar mechanisms to coreos-cloudinit but will provide the ability to configure a machine before the first boot. By configuring the system early, problems like ordering around network configuration are more easily solved. Just like coreos-cloudinit, Ignition will also have the ability to mark services to start on boot and configure user accounts.

Ignition is still under heavy development, but we are hoping to be able to start shipping it in CoreOS in the next couple of months.

Participate!

We encourage all of you as users of our systems and to continue having conversations with us. Please share ideas and tell us about what is working well, what may not be working well, and how can continue to have a useful feedback loop. In the GitHub repos for each of these projects, you can find a CONTRIBUTING.md and ROADMAP.md which outlines how to get started and where the projects are going. Thank you to our contributors!

We will also have the replays of the talks available at a later date, which will include a demo of Ignition and more

App Container spec gains new support as a community-led effort

2015-05-04T00:00:00+00:00

Today is the inaugural CoreOS Fest, the community event for distributed systems and application containers. We at CoreOS are here to celebrate you – those who want to join us on a journey to secure the backend of the Internet and build distributed systems technologies to bring web scale architecture to any organization. We've come a long way since releasing our first namesake project, CoreOS Linux, in 2013, and as a company we now foster dozens of open source projects as we work together with the community to create the components necessary for this new paradigm in production infrastructure.

An important part of working with this community has been the development of the App Container spec (appc), which provides a definition on how to build and run containerized applications. Announced in December, the appc spec emphasizes application container security execution, portability and modularity. rkt, a container runtime developed by CoreOS, is the first implementation of appc.

As security and portability between stacks becomes central to the successful adoption of application containers, today appc has gained support from various companies in the community:

Google has furthered its support of appc by implementing rkt into Kubernetes and joining as a maintainer of appc
Apcera has announced an additional appc implementation called Kurma
Red Hat has assigned an engineer to participate as a maintainer of appc
VMware recently announced how they will contribute to appc and shipped rkt in Project Photon

In order to ensure the specification remains a community-led effort, the appc project has established a governance policy and elected several new community maintainers unaffiliated with CoreOS: initially, Vincent Batts of Red Hat, Tim Hockin of Google and Charles Aylward of Twitter. This new set of maintainers brings each of their own unique points of view and allows appc to be a true collaborative effort. Two of the initial developers of the spec from CoreOS, Brandon Philips and Jonathan Boulle, remain as maintainers, but now are proud to have the collective help of others to make the spec what it is intended to be: open, well-specified and developed by a community.

In the months after the launch of appc, we have seen the adoption and support behind a common application container specification grow quickly. These companies and individuals are coming together to ensure there is a well defined specification for application containers, providing guidelines to ensure security, openness and modularity between stacks.

Google furthers its support of appc by integrating rkt into Kubernetes

Today also marks support for appc arriving in the Kubernetes project, via the integration of rkt as a configurable container runtime for Kubernetes clusters.

"The first implementation of the appc specification into Kubernetes, through the support of CoreOS rkt, is an important milestone for the Kubernetes project," said Craig McLuckie, product manager and Kubernetes co-founder at Google. "Designed with cluster first management in mind, appc support enables developers to use their preferred container image through the same Google infrastructure inspired orchestration framework."

Kubernetes is an open source project introduced by Google to help organizations run their infrastructure in a similar manner to the internal infrastructure that runs Google Search, Gmail and other Google services. Today's announcement of rkt being integrated directly into Kubernetes means that users will have the ability to run ACIs, the image format defined in the App Container spec, and take advantage of rkt’s first-class support for pods. rkt’s native support for running Docker images means they can also continue to use their existing images.

Apcera’s new implementation of appc, Kurma

Also announced today is Kurma, a new implementation of appc by Apcera. Kurma is an execution environment for running applications in containers. Kurma provides a framework that allows containers to be managed and orchestrated beyond itself. Kurma joins a variety of implementations of the appc spec that have emerged in the last six months, such as Jetpack, an App Container runtime for FreeBSD, and libappc, a C++ library for working with containerized applications.

"Apcera has long been invested in secure container technology to power our core platform," said Derek Collison, founder and CEO of Apcera. "We are excited to bring our technology to the open source community and to partner with CoreOS on the future of appc."

Red Hat involvement as a maintainer of appc

Red Hat recently assigned an engineer to participate as a maintainer of appc. Bringing years of experience in container development and leadership in Docker, Kubernetes and the Linux community as a whole, they bring a unique skillset to the effort.

“The adoption of container technology is an exciting trend and one that we believe can have significant customer benefit,” said Matt Hicks, senior director, engineering, Red Hat. “But at the same time, fragmentation of approaches and formats runs the risk of undercutting the momentum. We are excited to be included as maintainers and will work to not only innovate, but also to help create stability for our customers that adopt containers.”

VMware’s continued support of appc

In April, VMware announced support for appc and shipped rkt in Project Photon™, making rkt available to VMware vSphere® and VMware vCloud® Air™ customers. VMware has been an early proponent of appc and is working closely with the community to push forward the spec.

Today VMware reaffirmed their commitment to appc, showing its importance as a community-wide specification.

“VMware supports appc today offering rkt to our customers as a container runtime engine,” said Kit Colbert, vice president and CTO, Cloud-Native Apps, VMware. “We will work with the appc community to address portability and security across platforms – topics that are top of mind for enterprises seeking to support application containers in their IT environments.”

Join the appc community effort

We welcome these new companies into the community and invite others to join the movement to bring forward a secure and portable container standard. Get involved by joining the appc mailing list and discussion on GitHub. We welcome the continued independent implementations of tools to be able to run the same container consistently.

Thank you to all who are coming out to CoreOS Fest. Please follow along with the event on Twitter @CoreOSFest and #CoreOSFest. For those who aren't able to make it in person, the talks will be recorded and available at a later date.

CoreOS Fest 2015 Guide

2015-04-29T00:00:00+00:00

CoreOS Fest 2015 is in less than a week, and we want to make sure that you’re ready! To ensure that you have everything you need in order to have the best two days, we’ve put together a CoreOS Fest Guide.

If you haven’t gotten a ticket, but plan on joining us, there are only a few remaining tickets so be sure to register now while they are available.

We wouldn’t be here today without the help from our wonderful sponsors. Thank you to Intel, Google, VMware, AWS, Rackspace, Chef, Project Calico, Sysdig, Mesosphere and Giant Swarm.

Location

CoreOS Fest is located at The Village at 969 Market St. (between 5th and 6th St.) in downtown San Francisco, right by the Powell St. BART station. For local parking options, please check for options here.

Badge Pick-Up Times

The registration desk is located on the top floor of The Village. When you walk in, head straight up the stairs to pick up your badge.

Monday, May 4:
8:00 a.m. - end of day
At the registration desk on the top floor

Tuesday, May 5:
8:00 a.m. - end of day
At the registration desk on the top floor

Breakfast and Lunch Details

Breakfast and lunch will be held on the top floor each day. Dietary restrictions? We’ve accommodated for most diets, but if you’re concerned that we won’t have something for your specific diet, we recommend packing a lunch.

Monday, May 4:
Breakfast: 8 a.m. - 9 a.m.
Lunch: 11:45 a.m. - 1:00 p.m.

Tuesday, May 5:
Breakfast: 8:30 a.m. - 9:30 a.m.
Lunch: 11:45 a.m. - 1:00 p.m.

After Party Details

Join us May 4 for our After Party on the top floor of The Village from 5:45 p.m. to 8:00 p.m. We’ll also share a drink and a goodbye on May 5 at 5 p.m. - 6 p.m. at the AWS Pop-Up Loft next door, at 925 Market St.

CoreOS Office Hours

Attendees may sign up for office hours through a link you’ll get in your attendee email. Since there is a limited number of spots, please look at the conference schedule before getting your office hours tickets. Paper office hours tickets will not need to be shown at any time during CoreOS Fest as long as you have your badge.

Questions?

Have questions or need help the day of the event? You can email us at fest@coreos.com.

A Few Things to Keep in Mind

Be on time

Unlike CS101, this is something you’ll want to wake up for. We promise to have breakfast — and more importantly, coffee — waiting for you.

Talks will be recorded

All talks will be recorded, so if you miss one, don’t worry! All videos will be posted on the Fest ‘15 site after the event.

Bring a bag

Here at CoreOS, we believe that there is such a thing as having too many conference tote bags. If you’ll need a bag, make sure to bring your own, and we'll spare you the bagception dilemma.

Charging and Wi-Fi

Wi-Fi is available at the venue, along with charging stations and outlets.

Come with questions

Some of the most influential developers in infrastructure will be there to tell stories of their successes, missteps and lessons learned. They’re here to answer your questions, so bring on the tough ones!

Follow #CoreOSFest on Twitter

Make sure that you follow @CoreOSFest and #CoreOSFest on Twitter for live schedule updates, recorded talks and news.

We’re only a few days away from CoreOS Fest and we’re excited to see you all there!

Announcing GovCloud support on AWS

2015-04-27T00:00:00+00:00

Today we are happy to announce CoreOS Linux now supports Amazon Web Services GovCloud (US). AWS GovCloud is an isolated AWS Region for US government agencies and customers to move sensitive workloads into the AWS cloud by addressing their specific regulatory and compliance requirements. With this, automatic updates are now stable and available to all government agencies using the cloud.

CoreOS Linux customers will benefit from the security thanks to support of FedRAMP, a US government program sharing a standardized approach to security assessment, authorization and continuous monitoring for cloud products and services.

For more details, see the documentation on Running CoreOS on EC2.

rkt 0.5.4, featuring repository authentication, port forwarding and more

2015-04-24T00:00:00+00:00

Since the last rkt release a few weeks ago, development has continued apace, and today we're happy to announce rkt v0.5.4. This release includes a number of new features and improvements across the board, including authentication for image fetching, per-application arguments, running from pod manifests, and port forwarding support – check below the break for more details.

rkt, a container runtime for application containers, is under heavy development but making rapid progress towards a 1.0 release. Earlier this week, VMware announced support for rkt and the emerging App Container (appc) specification. appc is an open specification defining how applications can be run in containers, and rkt is the first implementation of the spec. With increasing industry commitment and involvement in appc, it is quickly fulfilling its promise of becoming a standard of how applications should be deployed in containers.

VMware released a short demo about how its new Project Photon works with rkt via Vagrant and VMware Fusion.

Read on below for more about the latest features in rkt 0.5.4.

Authentication for image fetching

rkt now supports HTTP Basic and OAuth Bearer Token authentication when retrieving remote images from HTTP endpoints and Docker registries. To facilitate this, we've introduced a flexible configuration system, allowing vendors to ship default configurations and then systems administrators to supplement or override configuration locally. Configuration is fully versioned to support forwards and backwards compatibility – check out the rkt documentation for more details.

Here's a simple example of fetching an image from a private Docker registry (note that Docker registries support only Basic authentication):

$ sudo cat /etc/rkt/auth.d/myuser.json 
{
    "rktKind": "dockerAuth",
    "rktVersion": "v1",
    "registries": ["quay.io"],
    "credentials": {
        "user": "myuser",
        "password": "sekr3tstuff"
    }
}
$ sudo /rkt --insecure-skip-verify fetch docker://quay.io/myuser/privateapp
rkt: fetching image from docker://quay.io/myuser/privateapp
Downloading layer: cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff
Downloading layer: 6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea
....

Per-application arguments and image signature verification for local images

The flag parsing in rkt run has been reworked to support per-app flags when running a pod with multiple images. Furthermore, in keeping with our philosophy of "secure by default", rkt will now attempt signature verification even when referencing local image files (during rkt fetch or rkt run commands). In this case, rkt expects to find the signature file alongside the referenced image – for example:

 $ rkt run imgs/pauser.aci
     error opening signature file: open /home/coreos/rkt/imgs/pauser.aci.asc: no such file or directory
 $ gpg2 --armor --detach-sign imgs/pauser.aci
 $ rkt run imgs/pauser.aci
     rkt: signature verified:
       Irma Bot (ACI Signing Key)
     ^]^]^]Container rootfs terminated by signal KILL.

Specific signatures can be provided with the --signature flag, which also applies per-app in the case of multiple references. In this example, we import two local images into the rkt CAS, specifying images signatures for both:

     $ rkt fetch   \
        imgs/pauser.aci --signature ./present.asc  \
        imgs/bash.aci --signature foo.asc
      rkt: signature verified:
        Joe Packager (CoreOS)
sha512-b680fd853abeba1a310a344e9fbf8ac9
sha512-ae78000a3d38fae4009699bf7494b293

Running from pod manifests

In previous versions of rkt, the arguments passed to rkt run (or rkt prepare) would be used to internally generate a Pod Manifest which is executed by later stages of rkt. This release introduces a new flag, --pod-manifest, to both rkt prepare and rkt run, to supply a pre-created pod manifest to rkt.

A pod manifest completely defines the execution environment of the pod to be run, such as volume mounts, port mappings, isolators, etc. This allows users complete control over all of these parameters in a well-defined way, without the need of a complicated rkt command-line invocation. For example, when integrating rkt as a container runtime for a cluster orchestration system like Kubernetes, the system can now programmatically generate a pod manifest instead of feeding a complicated series of arguments to the rkt CLI.

In this first implementation — and following the prescriptions of the upstream appc spec — the pod manifest is treated as the definitive record of the desired execution state: anything specified in the app fields will override what is in the original image, such as exec parameters, volumes mounts, port mappings, etc. This allows the operator to completely control what will be executed by rkt. Since the pod manifest is treated as a complete source of truth — and expected to be generated by orchestration tools with complete knowledge of the execution environment – --pod-manifest is initially considered mutually exclusive with other flags, such as --volumes and --port. See rkt run --help for more details.

Port forwarding

rkt now supports forwarding ports from the host to pods when using private networking.

As a simple example, given an app with the following ports entry in its Image Manifest:

{
    "name": "http",
    "port": 80,
    "protocol": "tcp"
}

the following rkt run command can be used to forward traffic from the host's TCP port 8888 to port 80 inside the pod:

rkt run --private-net --port=http:8888 myapp.aci

Whenever possible, it is more convenient to use a SDN solution like flannel to assign routable IPs to rkt pods. However, when such an option is not available, or for "edge" apps that require straddling both SDN and external networks (such as a load balancer), port forwarding can be used to expose select ports to the pod.

Testing, forward-compatibility, and more

There's plenty more under the hood in this release, including an extensive functional test harness, a new database schema migration process, and various internal improvements to the codebase. As we've talked about previously, rkt is a young project and we aren't yet able to guarantee API/ABI stability between releases, but forward-compatibility is a top priority for the forthcoming 0.6 release, and these changes are important steps towards this goal.

For full details of all the changes in this release, check out the release on GitHub.

Get involved!

We're on a journey to create an efficient, secure and composable application container runtime for production environments, and we want you to join us. Take part in the discussion through the rkt-dev mailing list or GitHub issues — and for those eager to get stuck in, contribute directly to the project. Are you doing interesting things with rkt or appc and want to share it with the world? Contact our marketing team at press@coreos.com.

VMware Ships rkt and Supports App Container Spec

2015-04-20T00:00:00+00:00

Today VMware shipped rkt, the application container runtime, and made it available to VMware customers in Project Photon. VMware also announced their support of the App Container spec, of which rkt is the first implementation.

“VMware is happy to provide rkt to offer our customers application container choice. rkt is the first implementation of the App Container spec (appc), and we look forward to contributing to the appc community to advance security and portability between platforms.”

— Kit Colbert, vice president and CTO, Cloud-Native Apps, VMware

We are thrilled to welcome VMware into the appc and rkt communities. The appc specific was formed to create an industry standard of how applications should be deployed in containers, with a focus on portability, composability, and security. rkt is a project originated by CoreOS to provide a production-ready Linux implementation of the specification.

VMware's extensive experience with running applications at scale in enterprise environments will be incredibly valuable as we work together with the community towards a 1.0 release of the appc specification and the rkt project.

Join us on our mission to create a secure, composable and standards-based container runtime. We welcome your involvement and contributions to rkt and appc:

join the ongoing discussion around the appc spec through the open issues on GitHub
reach out on the rkt and appc mailing lists
to jump right in, check out the "Help Wanted" label on the rkt GitHub project

etcd 2.0 in CoreOS Alpha Image

2015-04-16T00:00:00+00:00

Today we are pleased to announce that the first CoreOS image to have an etcd v2.0 release is now available in CoreOS alpha channel. etcd v2.0 marks a milestone in the evolution of etcd and includes many new features and improvements over etcd 0.4 including:

Reconfiguration protocol improvements: guards against accidental misconfiguration
New raft implementation: provides improved cluster stability
On-disk safety improvements: utilizes CRC checksums and append-only log behavior

etcd is an open source, distributed, consistent key-value store. It is a core component of CoreOS software that facilitates safe automatic updates, coordinates work scheduled to hosts, and sets up overlay networking for containers. Check out the etcd v2.0 announcement for more details on etcd and the new features.

We’ve been using etcd v2.0 in production behind discovery.etcd.io and quay.io for a few months now and it has proven to be stable in these use cases. All existing applications that use the etcd API should work against this new version of etcd. We have tested etcd v2.0 with applications like fleet, locksmith and flannel. The user facing API to etcd should provide the same features it had in the past; if you find issues please report them on GitHub.

Setup Using cloud-init

If you want to dive right in and try out bootstrapping a new cluster, the cloud-init docs have full details on all of the parameters. To support the new features of etcd v2.0, such as multiple listen addresses and proxy modes, a new cloud-init section named etcd2 is used. With a few lines of configuration and a new discovery token, you can take etcd v2.0 for a spin on your cluster.

IANA Ports

With the release of etcd2, we’ve taken the opportunity to begin the transition to our IANA-assigned port numbers: 2379 and 2380. For backward compatibility, etcd2 is configured to listen on both the new and old port numbers (4001 and 7001) by default, but this can always be further restricted as desired.

Migration and Changes

Existing clusters running etcd 0.4 clusters will not automatically migrate to etcd v2.0. As there are semantic changes in how etcd clusters are managed between the two versions, we have decided to include both. There are documented methods to migrate to etcd v2.0 and you may do this at your own pace. We encourage users to use etcd v2.0 for all new clusters to take advantage of the large number of stability and performance improvements over the older series.

In this process, we have had to break backward compatibility in two cases in order to support this change:

Starting fleet.service without explicitly starting etcd.service or etcd2.service will no longer work. If you are using fleet and need a local etcd endpoint, you will need to also start etcd.service or etcd2.service.
Starting flannel.service without explicitly starting etcd.service or etcd2.service will no longer work. If you are using flannel and need a local etcd endpoint, you will need to also start etcd.service or etcd2.service.

We have discouraged the use of this implicit dependency via our documentation but you can check if you will be affected. Make sure that etcd.service or etcd2.service are enabled or started in your cloud-config.

Looking Forward

As we look forward to etcd v2.1.0 and beyond, there are a number of exciting things shaping up inside of etcd. In the near future new features such as the authorization and authentication API will make it safer to operate multiple applications on a single cluster. The team has also been operating both on-going test environments that introduce regular partitions and crashes and making practical benchmarks available. In the last few days there has also been an active discussion on how to evolve the etcd APIs to better support the applications using etcd for coordination and scheduling today.

We welcome your involvement in the development of etcd - via the etcd-dev discussion mailing list, GitHub issues, or contributing directly to the project.

CoreOS on ARM64

2015-04-14T00:00:00+00:00

This is a guest post from CoreOS contributor, Geoff Levand, Linux Architect, Huawei America Software Lab. He has started work on an ARM64 port of CoreOS. Here is the current state of the project, followed by how you can help.

Recent patches that I've contributed to CoreOS have added basic support for a new target board named arm64-usr. There is currently a single generic ARM64 little endian Linux profile. This profile should work with any ARM64 platform currently supported by the mainline Linux kernel, so the ARM V8 Foundation Model, the ARM FVP_VE Fast Model, the ARM FVP_BASE Fast Model, and recent qemu-system-aarch64. I hope to add other profiles to support an ARM64 big endian build, and also to get the eight-core HiSilicon 6220 based HiKey developer board supported.

ARM64 porting work is still in progress, so please consider what is done so far as experimental. Some initial work I did along with Michael Marineau of CoreOS was to clean up parts of the CoreOS build system to simplify the way architectures are defined, and also to make the generic build infrastructure completely architecture agnostic. The resulting system should make it quite straight forward to add additional architecture support to CoreOS.

The ARM64 architecture is a relatively new one, so many upstream software packages have either only recently been updated to support ARM64, or have not yet been. Much of my CoreOS porting work so far has been going through the packages which don't build and figuring out how to get them to build. Sometimes a package can be updated to the latest upstream, sometimes a package keyword can be set, sometimes a modification to the ebuild in coreos-overlay will work, and other times a combination of these are needed. This process is still ongoing, and some difficult packages still lay ahead. The resulting arm64-usr build is experimental and all the work to bring it up will need testing and review in the future.

There is still a lot of work to be done. Many more packages need to be brought up, and as I mentioned, this involves working at a low level with the package ebuild files and the CoreOS build system. At another level, all the CoreOS features will need to be exercised and verified as needed to bring up the stability and confidence of the port. There are going to be multi-arch clusters, so ARM64 and x86_64 nodes are going to need to work together -- it sounds pretty cool. Someone will need to get in there and make that happen. If you have any interest in the ARM64 port I encourage you get involve and help out.

For general info about the port you can look at my Github site. For those who would like to investigate more, or even help with the effort, see my CoreOS ARM64 HOWTO document.

Continue the discussion with Geoff at CoreOS Fest and on freenode in #coreos as geoff-

Counting Down to CoreOS Fest on May 4 and 5

2015-04-13T00:00:00+00:00

As we count down to the inaugural CoreOS Fest in just three weeks, we are thrilled to announce additional speakers and the agenda! CoreOS Fest will be May 4-5 at The Village at 969 Market Street in San Francisco and we hope you will join us.

CoreOS Fest is a two-day event about the tools and best practices used to build modern infrastructure stacks. CoreOS Fest connects people from all levels of the community with future-thinking industry veterans to learn how to build distributed systems that support application containers. This May’s festival is brought to you by our premier sponsor Intel, and additional sponsors Sysdig, Chef, Mesosphere, Metaswitch Networks and Giant Swarm.

CoreOS Fest will include speakers from Google, Intel, Salesforce Data.com, HP, and more, including:

Brendan Burns, software engineer at Google and founder of Kubernetes, will provide a technical overview of Kubernetes
Diego Ongaro, creator of Raft, will discuss the Raft Consensus Algorithm
Lennart Poettering, creator of systemd, will talk about systemd at the Core of the OS
Nicholas Weaver, director of SDI-X at Intel, will demonstrate how we can optimize container architectures for the next level of scale
Prakash Rudraraju, manager of technical operations at Salesforce Data.com, will join Brian Harrington, principal architect at CoreOS, for a fireside chat on how Salesforce Data.com is thinking about distributed systems and application containers
Yazz Atlas, HPCS principle engineer with Hewlett-Packard Advanced Technology Group, will give a presentation on automated MySQL Cluster Failover using Galera Cluster on CoreOS Linux
Loris Degioanni, CEO and founder of Sysdig and co-creator of Wireshark, will present the dark art of container monitoring
Gabriel Monroy, CTO at OpDemand/Deis, will discuss lessons learned from building platforms on top of CoreOS
Spencer Kimball, founder of Cockroach Labs, will talk about CockroachDB
Chris Winslett, product manager at Compose.io, will present etcd based Postgres SQL HA Cluster
Timo Derstappen, co-founder of Giant Swarm, will present Containers on the Autobahn

More speakers will be added at https://coreos.com/fest/.

As a part of today's schedule announcement, we are offering 10 percent off the regular ticket price until tomorrow, April 14, at 10 a.m. PT. Use this link to reserve your 10 percent off ticket. Tickets are selling fast so get them before we sell out!

Once again, CoreOS Fest thanks its top level sponsor Intel and additional sponsors, including Sysdig, Chef, Mesosphere, Metaswitch Networks and Giant Swarm. If you’re interested in participating at CoreOS Fest as a sponsor, contact fest@coreos.com.

For more CoreOS Fest news, follow along @coreoslinux or #CoreOSFest

Upcoming CoreOS Events in April

2015-04-07T00:00:00+00:00

Supplied with fresh CoreOS t-shirts and half our weight in airport Cinnabons, we’ve made sure that you’ll be seeing a lot of us this April.

Wednesday, April 8, 2015 at 10:15 a.m. EDT - Philadelphia, PA

Don’t miss Kelsey Hightower (@kelseyhightower), developer advocate and toolsmith at CoreOS, kick off our April events by speaking at ETE Conference. He’ll be discussing managing containers at scale with CoreOS and Kubernetes.

Thursday, April 16, 2015 at 7:00 p.m. CET - Amsterdam, Netherlands

Kelsey Hightower will be giving an introduction to fleet, CoreOS and building large reliable systems at the Docker Randstad Meetup.

Thursday, April 16, 2015 at 6:00 p.m. PDT - San Francisco, CA

Brian Harrington will be giving an overview of CoreOS at CloudCamp. This is an unconference dedicated to all things containers.

Friday, April 17, 2015 - San Francisco, CA

Joined by a few of our very own, CoreOS CTO Brandon Philips (@BrandonPhilips) will be speaking at Container Camp. This event focuses on the latest developments in software virtualization. Get your tickets here.

Tuesday April 21 - Saturday, April 25, 2015 - Berlin, Germany

This year we’ll be attending Open Source Data Center Conference (OSDC) where Kelsey Hightower will be talking on building distributed systems with CoreOS.

Wednesday, April 22 at 6:30p.m. CET - Berlin, Germany

If you’re in Berlin, be sure to check out Kelsey Hightower talk about managing containers at scale with CoreOS and Kubernetes.

In case you missed it

In case you missed it, check out Chris Winslett from Compose.io talk about an etcd-based PostgreSQL HA Cluster:

CoreOS Fest

Don’t forget that CoreOS Fest is happening the following month on May 4 and 5! We’ve released a tentative schedule and our first round of speakers. Keep checking back for more updates as the event gets closer.

Announcing Tectonic: The Commercial Kubernetes Platform

2015-04-06T00:00:00+00:00

Our technology is often characterized as “Google’s infrastructure for everyone else.” Today we are excited to make this idea a reality by announcing Tectonic, a commercial Kubernetes platform. Tectonic provides the combined power of the CoreOS portfolio and the Kubernetes project to any cloud or on-premise environment.

Why we are building Tectonic

Our users want to securely run containers at scale in a distributed environment. We help companies do this by building open source tools which allow teams to create this type of infrastructure. With Tectonic, we now have an option for companies that want a preassembled and enterprise-ready distribution of these tools, allowing them to quickly see the benefits of modern container infrastructure.

What is Tectonic?

Tectonic is a platform combining Kubernetes and the CoreOS stack. Tectonic pre-packages all of the components required to build Google-style infrastructure and adds additional commercial features, such as a management console for workflows and dashboards, an integrated registry to build and share Linux containers, and additional tools to automate deployment and customize rolling updates.

Tectonic is available today to a select number of early customers. Head over to tectonic.com to sign up for the waitlist if your company is interested in participating.

What is Kubernetes?

Kubernetes is an open source project introduced by Google to help organizations run their infrastructure in a similar manner to the internal infrastructure that runs Google Search, Gmail, and other Google services. The concepts and workflows in Kubernetes are designed to help engineers focus on their application instead of infrastructure and build for high availability of services. With the Kubernetes APIs, users can manage application infrastructure - such as load balancing, service discovery, and rollout of new versions - in a way that is consistent and fault-tolerant.

Tectonic and CoreOS

Tectonic is a commercial product, and with this release, we have decided to launch our commercial products under a new brand, separate from the CoreOS name. We want our open source components - like etcd, rkt, flannel, and CoreOS Linux - to always be freely available for everyone under their respective open source licenses. We think open source development works best when it is community-supported infrastructure that we all share and build with few direct commercial motives. To that end, we want to keep CoreOS focused on building completely open source components.

To get access to an early release of Tectonic or to learn more, visit tectonic.com. To contribute and learn more about our open source projects visit coreos.com.

Google Ventures Funding

In addition to introducing Tectonic, today we are announcing an investment in CoreOS, Inc. led by Google Ventures. It is great to have the support and backing of Google Ventures as we bring the Kubernetes platform to market. The investment will help us accelerate our efforts to secure the backend of the Internet and deliver Google-like infrastructure to everyone else.

FAQ

Q: What does this change about CoreOS Linux and other open source projects like rkt, etcd, fleet, flannel, etc?

A: Nothing: development will continue, and we want to see all of the open source projects continue to thrive as independent components. CoreOS Linux will remain the same carefully maintained, open source, and container-focused OS it has always been. Tectonic uses many of these projects internally - including rkt, etcd, flannel, and fleet - and runs on top of the same CoreOS Linux operating system as any other application would.

Q: I am using Apache Mesos, Deis, or another application on top of CoreOS Linux: does anything change for me?

A: No, this announcement doesn't change anything about the CoreOS Linux project or software. Tectonic is simply another container-delivered application that runs on top of CoreOS Linux.

Q: What does this change for existing Enterprise Registry, Managed Linux, or Quay.io customers?

A: Everything will remain the same for existing customers. All of these components are utilized in the Tectonic stack and we continue to offer support, fix bugs and add features to these products.

Follow @TectonicStack on Twitter

Go to Tectonic.com to join an early release or to stay up to date on Tectonic news

Visit us in person at CoreOS Fest in San Francisco May 4-5, to learn more about CoreOS, Tectonic and all things distributed systems

Announcing rkt v0.5, featuring pods, overlayfs, and more

2015-04-01T00:00:00+00:00

rkt is a new container runtime for applications, intended to meet the most demanding production requirements of security, efficiency and composability. rkt is also an implementation of the emerging Application Container (appc) specification, an open specification defining how applications can be run in containers. Today we are announcing the next major release of rkt, v0.5, with a number of new features that bring us closer to these goals, and want to give an update on the upcoming roadmap for the rkt project.

appc v0.5 - introducing pods

This release of rkt updates to the latest version of the appc spec, which introduces pods. Pods encapsulate a group of Application Container Images and describe their runtime environment, serving as a first-class unit for application container execution.

Pods are a concept recently popularised by Google's Kubernetes project. The idea emerged from the recognition of a powerful, pervasive pattern in deploying applications in containers, particularly at scale. The key insight is that, while one of the main value propositions of containers is for applications to run in isolated and self-contained environments, it is often useful to co-locate certain "helper" applications within a container. These applications have an intimate knowledge of each other - they are designed and developed to work co-operatively - and hence can share the container environment without conflict, yet still be isolated from interfering with other application containers on the same system.

A classic example of a pod is service discovery using the sidekick model, wherein the main application process serves traffic, and the sidekick process uses its knowledge of the pod environment to register the application in the discovery service. The pod links together the lifecycle of the two processes and ensures they can be jointly deployed and constrained in the cluster.

Another simple example is a database co-located with a backup worker. In this case, the backup worker could be isolated from interfering with the database's work - through memory, I/O and CPU limits applied to the process - but when the database process is shut down the backup process will terminate too. By making the backup worker an independent application container, and making pods the unit of deployment, we can reuse the worker for backing up data from a variety of applications: SQL databases, file stores or simple log files.

This is the power that pods provide: they encapsulate a self-contained, deployable unit that still provides granularity (for example, per-process isolators) and facilitates advanced use cases. Bringing pods to rkt enables it to natively model a huge variety of application use cases, and integrate tightly with cluster-level orchestration systems like Kubernetes.

For more information on pods, including the technical definition, check out the appc spec or the Kubernetes documentation.

overlayfs support

On modern Linux systems, rkt now uses overlayfs by default when running application containers. This provides immense benefits to performance and efficiency: start times for large containers will be much faster, and multiple pods using the same images will consume less disk space and can share page cache entries.

If overlayfs is not supported on the host operating system, rkt gracefully degrades back to the previous behaviour of extracting each image at runtime - this behaviour can also be triggered with the new --no-overlay flag to rkt run.

Another improvement behind the scenes is the introduction of a tree cache for rkt's local image storage. When storing ACIs in its local database (for example, after pulling them from a remote repository using rkt fetch), rkt will now store the expanded root filesystem of the image on disk. This means that when pods that reference this image are subsequently started (via rkt run), the pod filesystem can be created almost instantaneously in the case of overlayfs - or, without overlayfs, by using a simple copy instead of needing to expand the image again from its compressed format.

To facilitate simultaneous use of the tree store by multiple rkt invocations, file-based locking has been added to ensure images that are in use cannot be removed. Future versions of rkt will expose more advanced capabilities to manage images in the store.

stage1 from source

When executing application containers, rkt uses a modular approach (described in the architecture documentation) to support swappable, alternative execution environments. The default stage1 that we develop with rkt itself is based on systemd, but alternative implementations can leverage different technologies like KVM-based virtual machines to execute applications.

In earlier versions of rkt, the pre-bundled stage1 was assembled from a copy of the CoreOS Linux distribution image. We have been working hard to decouple this process to make it easier to package rkt for different operating systems and in different build environments. In rkt 0.5, the default stage1 is now constructed from source code, and over the next few releases we will make it easier to build alternative stage1 images by documenting and stabilizing the ABI.

"Rocket", "rocket", "rkt"?

This release also sees us standardizing on a single name for all areas of the project - the command-line tool, filesystem names and Unix groups, and the title of the project itself. Instead of "rocket", "Rocket", or "rock't", we now simply use "rkt".

Looking forward

rkt is a young project and the last few months have seen rapid changes to the codebase. As we look towards rkt 0.6 and beyond, we will be focusing on making it possible to depend on rkt to roll-forward from version to version without breaking working setups. There are several areas that are needed to make this happen, including reaching the initial stable version (1.0) of the appc spec, implementing functional testing, stabilizing the on-disk formats, and implementing schema upgrades for the store. We realize that stability is vital for people considering using rkt in production environments, and this will be a priority in the next few releases. The goal is to make it possible for a user that was happily using rkt 0.6 to upgrade to rkt 0.7 without having to remove their downloaded ACIs or configuration files.

We welcome your involvement in the development of rkt - via the rkt-dev discussion mailing list, GitHub issues, or contributing directly to the project.

CoreOS Fest 2015 First Round of Speakers Announced

2015-03-27T00:00:00+00:00

As you might already know, we’re launching our first ever CoreOS Fest this May 4th and 5th in San Francisco! We’ve been hard at work making sure that this event is two days filled with all things distributed, and all things awesome.

In addition to many CoreOS project leads taking the stage, we are excited to announce a sneak peek at some of our community speakers. Join us at CoreOS Fest and you’ll hear from some of the most influential people in distributed systems today: Brendan Burns, one of the founders of Kubernetes; Diego Ongaro, the creator of Raft; Gabriel Monroy, the creator of Deis; Spencer Kimball, CEO of Cockroach Labs; Loris Degioanni, CEO of Sysdig; and many more!

We are still accepting submissions for speakers through March 31st, so we encourage you to submit your talk in our Call for Papers portal.

While the schedule will be live in the coming weeks, here's a high level overview:

We’ll kick off day one at 9 AM PDT (with registration and breakfast beforehand) with a single track of speakers, followed by lunch, then afternoon panels and breakouts. You’ll have lots of opportunities to connect and talk with fellow attendees, especially at an evening reception on the first day. Day two will include breakfast, single-track talks, lunch, panels and more.

Confirmed Speakers

See more about our first round of speakers:

Brendan Burns

Software Engineer at Google and a founder of the Kubernetes project

Brendan works in the Google Cloud Platform, leading engineering efforts to make the Google Cloud Platform the best place to run containers. He also has managed several other cloud teams including the Managed VMs team, and Cloud DNS. Prior to Cloud, he was a lead engineer in Google’s web search infrastructure, building backends that powered social and personal search. Prior to working at Google, he was a professor at Union College in Schenectady, NY. He received a PhD in Computer Science from the University of Massachusetts Amherst, and a BA in Computer Science and Studio Art from Williams College.

Diego Ongaro

Creator of Raft

Diego recently completed his doctorate with John Ousterhout at Stanford. During his doctorate, he worked on RAMCloud (a 5-10 microsecond RTT key-value store), Raft, and LogCabin (a coordination service built with Raft). He’s lately been continuing development on LogCabin as an independent contractor.

Gabriel Monroy

CTO of OpDemand and creator of Deis

Gabriel Monroy is CTO at OpDemand and the creator of Deis, the leading CoreOS-based PaaS. As an early contributor to Docker and CoreOS, Gabriel has deep experience putting containers into production and frequently advises organizations on PaaS, container automation and distributed systems. Gabriel spoke recently at QConSF on cluster scheduling and deploying containers at scale.

Spencer Kimball

CEO of Cockroach Labs

Spencer is CEO of Cockroach Labs. After helping to re-architect and re-implement Square's items catalog service, Spencer was convinced that the industry needed a more capable database software. He began work on the design and implementation of Cockroach as an open source project and moved to work on it full time at Square mid-2014. Spencer managed the acquisition of Viewfinder by Square as CEO and before that, shared the roles of co-CTO and co-founder. Previously, he worked at Google on systems and web application infrastructure, most recently helping to build Colossus, Google’s exascale distributed file system, and on Java infrastructure, including the open-sourced Google Servlet Engine.

Loris Degioanni

CEO of Sysdig

Loris is the creator and CEO of Sysdig, a popular open source troubleshooting tool for Linux environments. He is a pioneer in the field of network analysis through his work on WinPcap and Wireshark: open source tools with millions of users worldwide. Loris was previously a senior director of technology at Riverbed, and co-founder/CTO at CACE Technologies, the company behind Wireshark. Loris holds a PhD in computer engineering from Politecnico di Torino, Italy.

Excited? Stay tuned for more announcements and join us at CoreOS Fest 2015.

Buy your early bird ticket by March 31st: https://coreos.com/fest/

Submit a speaking abstract by March 31st: CFP Portal

Become a sponsor, email us for more details.

What makes a cluster a cluster?

2015-03-20T00:00:00+00:00

“What makes a cluster a cluster?” - Ask that question of 10 different engineers and you’ll get 10 different answers. Some look at it from a hardware perspective, some see it as a particular set of cloud technologies, and some say it’s the protocols exchanging information on the network.

With this ever-growing field of distributed systems technologies, it is helpful to compare the goals, roles and differences of some of these new projects based on their functionality. In this post we propose a conceptual description of the cluster at large, while showing some examples of emerging distributed systems technologies.

Layers of abstraction

The tech community has long agreed on what a network looks like. We’ve largely come to agree, in principle, on the OSI (Open Systems Interconnection) model (and in practice, on its close cousin, the TCP/IP model).

A key aspect of this model is the separation of concerns, with well-defined responsibilities and dependence between components: every layer depends on the layer below it and provides useful network functionality (connection, retry, packetization) to the layer above it. At the top, finally, are web sessions and applications of all sorts running and abstracting communication.

So, as an exercise to try to answer “What makes a cluster a cluster?” let’s apply the same sort of thinking to layers of abstraction in terms of execution of code on a group of machines, instead of communication between these machines.

Here’s a snapshot of the OSI model, applied to containers and clustering:

Let’s take a look from the bottom up.

Level 1, Hardware

The hardware layer is where it all begins. In a modern environment, this may mean physical (bare metal) or virtualized hardware – abstraction knows no bounds – but for our purposes, we define hardware as the CPU, RAM, disk and network equipment that is rented or bought in discrete units.

Examples: bare metal, virtual machines, cloud

Level 2, OS/Machine ABI

The OS layer is where we define how software executes on the hardware: the OS gives us the Application Binary Interface (ABI) by which we agree on a common language that our userland applications speak to the OS (system calls, device drivers, and so on). We also set up a network stack so that these machines can communicate amongst each other. This layer therefore provides our lowest level complete execution environment for applications.

Now, traditionally, we stop here, and run our final application on top of this as a third pseudo-layer of the OS and various user-space packages. We provision individual machines with slightly different software stacks (a database server, an app server) and there’s our server rack.

Over the lifetime of servers and software, however, the permutations and histories of individual machine configurations start to become unwieldy. As an industry, we are learning that managing this complexity becomes costly or infeasible over time, even at moderate scale (e.g. 3+ machines).

This is often where people start to talk about containers, as containers treat the entire OS userland as one hermetic application package that can be managed as an independent unit. Because of this abstraction, we can conceptually shift containers up the stack, as long as they’re above layer 2. We’ll revisit containers in layer 6.

Examples: kernel + {systemd, cgroups/namespaces, jails, zones}

Level 3, Cluster Consensus

To begin to mitigate the complexity of managing individual servers, we need to start thinking about machines in some greater, collective sense: this is our first notion of a cluster. We want to write software that scales across these individual servers and shares work effortlessly.

However, as we add more servers to the picture, we now introduce many more points of failure: networks partition, machines crash and disks fail. How can we build systems in the face of greater uncertainty? What we’d like is some way of creating a uniform set of data and data primitives, as needed by distributed systems. Much like in multiprocessor programming, we need the equivalent of locks, message passing, shared memory and atomicity across this group of machines.

This is an interesting and vibrant field of algorithmic research: a first stop for the curious reader should be the works of Leslie Lamport, particularly his earlier writing on ordering and reliability of distributed systems. His later work describes Paxos, the preeminent consensus protocol; the other major protocol, as provided by many projects in this category, is Raft.

Why is this called consensus? The machines need to ‘agree’ on the same history and order of events in order to make the guarantees we’d like for the primitives described. Locks cannot be taken twice, for example, even if some subset of messages disappears or arrives out of order, or member machines crash for unknown reasons.

These algorithms build data structures to form a coherent, consistent, and fault-tolerant whole.

Examples: etcd, ZooKeeper, consul

Level 4, Cluster Resources

With this perspective of a unified cluster, we can now talk about cluster resources. Having abstracted the primitives of individual machines, we use this higher level view to create and interact with the complete set of resources that we have at our disposal. Thus we can consider in aggregate the CPUs, RAM, disk and networking as available to any process in the cluster, as provided by the physical layers underneath.

Viewing the cluster as one large machine, all devices (CPU, RAM, disk, networking) become abstract. This is a benefit already being used by containers. Containers depend on these things being abstracted on their behalf; for example, network bridges. This is so they can use these abstractions at a level higher in the stack while running on any of the underlying hardware.

In some sense, this layer is the equivalent of the hardware layer of the now-primordial notion of the cluster. It may not be as celebrated as the layers above it, but this layer is where some important innovation takes place. Showing a cool auto-scaling webapp demo is nice, but requires things like carving up cluster IP space or where a block device is attached to a host.

Examples: flannel, remote block storage, weave

Level 5, Cluster Orchestration and Scheduling

Cluster orchestration, then, starts to look a lot like an OS kernel atop these cluster-level resources and the tools given by consistency – symmetry with the layers below again. It’s the purview of the orchestration platform to divide and share cluster resources, schedule applications to run, manage permissions, set up interfaces into and out of the cluster, and at the end of the day, find an ABI-compatible environment for the userland. With increased scale comes new challenges: from finding the right machines to providing the best experience to users of the cluster.

Any software that will run on the cluster must ultimately execute on a physical CPU on a particular server. How the application code gets there and what abstractions it sees is controlled by the orchestration layer. This is similar to how WiFi simulates a copper wire to existing network stacks, with a controllable abstraction through access points, signal strength, meshes, encryption and more.

Examples: fleet, Mesos, Kubernetes

Level 6, Containers

This brings us back to containers, which, as described earlier, the entire userland is bundled together and treated as a single application unit.

If you’ve followed the whole stack up to this point, you’ll see why containers sit at level 6, instead of at level 2 or 3. It’s because the layers of abstraction below this point all depend on each other to build up to the point where a single-serving userland can safely abstract whether it’s running as one process on a local machine or as something scheduled on the cluster as a whole.

Containers are actually simple that way; they depend on everything else to provide the appropriate execution environment. They carry userland data and expect specific OS details to be presented to them.

Examples: Rocket, Docker, systemd-nspawn

Level 7, Application

Containers are currently getting a lot of attention in the industry because they can separate the OS and software dependencies from the hardware. By abstracting these details, we can create consistent execution environments across a fleet of machines and let the traditional POSIX userland continue to work, fairly seamlessly, no matter where you take it. If the intention is to share the containers, then choice is important, as is agreeing upon a sharable standard. Containers are exciting; it starts us down the road of a lot of open source work in the realm of true distributed systems, backwards-compatible with the code we already write – our Application.

Closing Thoughts

For any of the layers of the cluster, there are (and will continue to be) multiple implementations. Some will combine layers, some will break them into sub-pieces – but this was true of networking in the past as well (do you remember IPX? Or AppleTalk?).

As we continue to work deeply on the internals of every layer, we also sometimes want to take a step back to look at the overall picture and consider the greater audience of people who are interested and starting to work on clusters of their own. We want to introduce this concept as a guideline, with a symmetric way of thinking about a cluster and its components. We’d love your thoughts on what defines a cluster as more than a mass of hardware.

Announcing rkt and App Container 0.4.1

2015-03-13T00:00:00+00:00

Today we are announcing rkt v0.4.1. rkt is a new app container runtime and implementation of the App Container (appc) spec. This milestone release includes new features like private networking, an enhanced container lifecycle, and unprivileged image fetching, all of which get us closer to our goals of a production-ready container runtime that is composable, secure, and fast.

Private Networking

This release includes our first iteration of the rkt networking subsystem. As an example, let's run etcd in a private network:

# Run an etcd container in a private network
$ rkt run --private-net coreos.com/etcd:v2.0.4

By using the --private-net flag, the etcd container will run with its own network stack decoupled from the host. This includes a private lo loopback device and an eth0 device with an IP in the 172.16.28.0/24 address range. By default, rkt creates a veth pair, with one end becoming eth0 in the container and the other placed on the host. rkt will also set up an IP masquerade rule (NAT) to allow the container to speak to the outside world.

This can be demonstrated by being able to reach etcd on its version endpoint from the host:

$ curl 172.16.28.9:2379/version
{"releaseVersion":"2.0.4","internalVersion":"2"}

The networking configuration in rkt is designed to be highly pluggable to facilitate a variety of networking topologies and infrastructures. In this release, we have included plugins for veth, bridge, and macvlan, and more are under active development. See the rkt network docs for details.

If you are interested in building new network plugins, please take a look at the current specification and get involved by reaching out on GitHub or the mailing list. We would also like to extend a thank you to everyone who has spent time giving valuable feedback on the spec so far.

Unprivileged Fetches

It is good practice to download files over the Internet only as unprivileged users. With this release of rkt, it is possible to set up a rkt Unix group, and give users in that group the ability to download and verify container images. For example, let's give the core user permission to use rkt to retrieve images and verify their signature:

$ sudo groupadd rkt
$ sudo usermod -a -G rkt core
$ sudo rkt install
$ rkt fetch coreos.com/etcd:v2.0.5
rkt: searching for app image coreos.com/etcd:v2.0.5
rkt: fetching image from https://github.com/coreos/etcd/releases/download/v2.0.5/etcd-v2.0.5-linux-amd64.aci
Downloading ACI: [==========                                   ] 897 KB/3.76 MB
Downloading signature from https://github.com/coreos/etcd/releases/download/v2.0.5/etcd-v2.0.5-linux-amd64.aci.asc
rkt: signature verified:                                       ] 0 B/819 B
  CoreOS ACI Builder <release@coreos.com>
sha512-295a78d35f7ac5cc919e349837afca6d

The new rkt install subcommand is a simple helper to quickly set up all of the rkt directory permissions. These steps could easily be scripted outside of rkt for a more complex setup or a custom group name; for example, distributions that package rkt in their native formats would configure directory permissions at the time the package is installed.

Note that the image we’ve fetched will still need to be run with sudo, as Linux doesn't yet make it possible to do many of the operations necessary to start a container without root privileges. But at this stage, you can trust that the image comes from an author you have already trusted via rkt trust.

Other Features

rkt prepare is a new command that can be used to set up a container without immediately running it. This gives users the ability to allocate a container ID and do filesystem setup before launching any processes. In this way, a container can be prepared ahead of time, so that when rkt run-prepared is subsequently invoked, the process startup happens immediately with few additional steps. Being able to pre-allocate a unique container ID also facilitates better integration with higher-level orchestration systems.

rkt run can now append additional command line flags and environment variables for all apps, as well as optionally have containers inherit the environment from the parent process. For full details see the command line documentation.

The image store now uses a ql database to track metadata about images in the store. This is used to keep track of URLs, labels, and other metadata of images stored inside rkt's local store. Note that if you are upgrading from a previous rkt release on a system, you may need to remove /var/lib/rkt. We understand people are already beginning to rely on rkt and over the next few releases will focus heavily on introducing stable APIs. But until we are closer to a 1.0 release, expect that there will be more regular changes.

For more details about this 0.4.1 release and pre-compiled standalone rkt Linux binaries see the release page.

Updates to App Container spec

Finally, this change updates rkt to the latest version of the appc spec, v0.4.1. Recent changes to the spec include reworked isolators, new OS-specific requirements, and greater explicitness around image signing and encryption. You can refer to a list of some major changes and additions here.

Join us on the mission to create a secure, composable and standards based container runtime, and get involved in hacking on rkt or App Container here:

rkt:

Help Wanted, Mailing list

App Container:

Help Wanted, Mailing list

rkt Now Available in CoreOS Alpha Channel

2015-03-12T00:00:00+00:00

Our CoreOS Alpha channel is designed to strike a balance between offering early access to new versions of software and serving as the release candidate for the Beta and Stable channels. Due to its release-candidate nature, we must be conservative in upgrading critical system components (e.g. systemd and etcd), but in order to get new technologies (like fleet and flannel) into the hands of users for testing we must occasionally include pre-production versions of these components in Alpha.

Today, we are adding rkt, a container runtime built on top of the App Container spec, to make it easier for users to try it and give us feedback.

rkt will join systemd-nspawn and Docker as container runtimes that are available to CoreOS users. Keep in mind that rkt is still pre-1.0 and that you should not rely on flags or the data in /var/lib/rkt to work between versions. Specifically, next week v0.4.1 will land in Alpha which is incompatible with images and containers created by previous versions of rkt. Besides the addition of /usr/bin/rkt to the image, nothing major has changed and no additional daemons will run by default.

Release Cadence

We have adopted a regular weekly schedule for Alpha releases, rolling out a new version every Thursday. Every other week we release a Beta, taking the best of the previous two Alpha versions and promoting it bit-for-bit. Similarly, once every four weeks we promote the best of the previous two Beta releases to Stable.

Give it a spin

If you want to spin up a CoreOS Alpha machine and get started, check out the documentation for v0.3.2. We look forward to having you involved in rkt development via the rkt-dev discussion mailing list, GitHub issues, or contributing directly to the project. We have made great progress so far, but there is still much to build!

The First CoreOS Fest

2015-03-11T00:00:00+00:00

Get ready, CoreOS Fest, our celebration of everything distributed, is right around the corner! Our first CoreOS Fest is happening May 4 and 5, 2015 in San Francisco. You’ll learn more about application containers, container orchestration, clustering, devops security, new Linux, Go and more.

Join us for this two-day event as we talk about the newest in distributed systems technologies and together talk about securing the Internet. Be part of discussions shaping modern infrastructure stacks, hear from peers on how they are using these technologies today and get inspired to learn new ways to speed up your application development process.

Take a journey with us (in space and time) and help contribute to the next generation of infrastructure. The early bird tickets are available until March 31st and are only $199, so snatch one up now before they are gone. After March 31st, tickets will be available for $349. See you in May.

Submit an Abstract

Grab An Early Bird Ticket

If you are interested in sponsoring the event, reach out to fest@coreos.com and we would be happy to send you the prospectus.

CoreOS on VMware vSphere and VMware vCloud Air

2015-03-09T00:00:00+00:00

At CoreOS, we want to make the world successful with containers on all computing platforms. Today, we are taking one step closer to that goal by announcing, with VMware, that CoreOS is fully supported and integrated with both VMware vSphere 5.5 and VMware vCloud Air. Enterprises that have been evaluating using containers but needed fully supported environments to begin now have the support to get started.

We’ve worked closely with VMware in enabling CoreOS to run on vSphere 5.5 (see the technical preview of CoreOS on vSphere 5.5). This collaboration extends the security, consistency, and reliability advantages of CoreOS to users of vSphere. Developers can focus on their applications and operations get the control they need. We encourage you to read more from VMware here:

CoreOS Now Supported on VMware vSphere 5.5 and VMware vCloud Air.

As a sysadmin you’ve gotta be thinking, what does this mean for me?

Many people have been running CoreOS on VMware for a while now, but something was missing. Mainly performance and full integration with VMware management APIs. Today that all changes. CoreOS is now shipping open-vm-tools, the open source implementation of VMware Tools, which enables better performance and enables management of CoreOS VMs running in all VMware environments.

Lets take a quick moment to explore some of the things that are now possible.

Taking CoreOS for a spin with VMware Fusion

The following tutorial will walk you through downloading an official CoreOS VMware image and configuring it using a cloud config drive. Once configured, a CoreOS instance will be launched and managed using the vmrun command line tool that ships with VMware Fusion.

To make the following commands easier to run set the following vmrun alias in your shell:

alias vmrun='/Applications/VMware\ Fusion.app/Contents/Library/vmrun'

Download a CoreOS VMware Image

First things first, download a CoreOS VMware image and save it to your local machine:

$ mkdir coreos-vmware
$ cd coreos-vmware
$ wget http://alpha.release.core-os.net/amd64-usr/current/coreos_production_vmware.vmx
$ wget http://alpha.release.core-os.net/amd64-usr/current/coreos_production_vmware_image.vmdk.bz2

Decompress the VMware disk image:

$ bzip2 -d coreos_production_vmware_image.vmdk.bz2

Configuring a CoreOS VM with a config-drive

By default CoreOS VMware images do not have any users configured, which means you won’t be able to login to your VM after it boots. Also, many of the vmrun guest OS commands require a valid CoreOS username and password.

A config-drive is the best way to configure a CoreOS instance running on VMware. Before you can create a config-drive, you’ll need some user data. For this tutorial you will use a CoreOS cloud-config file as user data to configure users and set the hostname.

Generate the password hash for the core and root users

Before creating the cloud-config file, generate a password hash for the core and root users:

$ openssl passwd -1
Password:
Verifying - Password:
$1$LEfVXsiG$lhcyOrkJq02jWnEhF93IR/

Enter vmware at both password prompts.

Create a cloud config file

Now we are ready to create a cloud-config file:

edit cloud-config.yaml

#cloud-config

hostname: vmware-guest
users:
  - name: core
    passwd: $1$LEfVXsiG$lhcyOrkJq02jWnEhF93IR/
    groups:
      - sudo
      - docker
  - name: root
    passwd: $1$LEfVXsiG$lhcyOrkJq02jWnEhF93IR/

Create a config-drive

With your cloud-config file in place you can use it to create a config drive. The easiest way to create a config-drive is to generate an ISO using a cloud-config file and attach it to a VM.

$ mkdir -p /tmp/new-drive/openstack/latest
$ cp cloud-config.yaml /tmp/new-drive/openstack/latest/user_data
$ hdiutil makehybrid -iso -joliet -joliet-volume-name "config-2" -o ~/cloudconfig.iso /tmp/new-drive
$ rm -r /tmp/new-drive

At this point you should have a config-drive named cloudconfig.iso in your home directory.

Attaching a config-drive to a VM

Before booting the CoreOS VM the config-drive must be attached to the VM. Do this by appending the following lines to the coreos_production_vmware.vmx config file:

ide0:0.present = "TRUE"
ide0:0.autodetect = "TRUE"
ide0:0.deviceType = "cdrom-image"
ide0:0.fileName = "/Users/kelseyhightower/cloudconfig.iso"

At this point you are ready to launch the CoreOS VM:

vmrun start coreos_production_vmware.vmx

Running commands

With the CoreOS VM up and running use the vmrun command line tool to interact with it. Let's start by checking the status of vmware-tools in the VM:

$ vmrun checkToolsState coreos_production_vmware.vmx

Grab the VM’s IP address with the getGuestIPAddress command:

$ vmrun getGuestIPAddress coreos_production_vmware.vmx

Full VMware integration also means you can now run guest OS commands. For example you can list the running processes using the listProcessesInGuest command:

$ vmrun -gu core -gp vmware listProcessesInGuest coreos_production_vmware.vmx
Process list: 63
pid=1, owner=root, cmd=/usr/lib/systemd/systemd --switched-root --system --deserialize 21
pid=2, owner=root, cmd=kthreadd
pid=3, owner=root, cmd=ksoftirqd/0
pid=4, owner=root, cmd=kworker/0:0
pid=5, owner=root, cmd=kworker/0:0H
pid=6, owner=root, cmd=kworker/u2:0
...

Finally you can now run arbitrary commands and scripts using VMware management tools. For example, use the runProgramInGuest command to initiate a graceful shutdown:

$ vmrun -gu root -gp vmware runProgramInGuest coreos_production_vmware.vmx /usr/sbin/shutdown now

We have only scratched the surface regarding the number of things you can do with the new VMware powered CoreOS images. Check out the “Using vmrun to Control Virtual Machines” e-book for more details.

CoreOS and VMware going forward

We look forward to continuing on the journey to secure the backend of the Internet by working on all types of platforms in the cloud or behind the firewall. We are continuing to work with VMware so that CoreOS is also supported on the recently announced vSphere 6. If you have any questions in the meantime, you can find us on IRC as you get started. Feedback can also be provided at the VMware / CoreOS community forum.

Managing CoreOS Logs with Logentries

2015-03-05T00:00:00+00:00

Today Logentries announced a CoreOS integration, so CoreOS users can get a a deeper understanding into their CoreOS environments. The new integration enables CoreOS users to easily send logs using the Journal logging system, part of CoreOS’ Systemd process manager, directly into Logentries for real-time monitoring, alerting, and data visualization. This is the first CoreOS log management integration.

To learn more about centralizing logs from CoreOS clusters read Trevor Parsons, co-founder and chief scientist at Logentires post. Or get started by following the documentation here.

Upcoming CoreOS Events in March

2015-03-03T00:00:00+00:00

March brings a variety of events – including a keynote from Alex Polvi (@polvi), CEO of CoreOS, at Rackspace Solve. Read on for more details on the team’s whereabouts this month.

In case you missed it, Alex keynoted at The Linux Foundation Collab Summit last month. See the replay.

Tuesday, March 3, 2015 at 6 p.m. EST – Montreal, QC

The month kicks off with Jake Moshenko (@JacobMoshenko), product manager for the Quay.io container registry at CoreOS, at Big Data Montreal. Jake will discuss Rocket and how CoreOS and Quay.io fit into the development lifecycle.

Wednesday, March 4, 2015 at 1:30 p.m. PST – San Francisco, CA

Join us at Rackspace Solve to see Alex Polvi (@polvi), CEO of CoreOS, speak about Container Technology: Applications and Implications. Registration is free and there will be talks from Wikimedia, Walmart Labs, DigitalFilmTree, Tinder and more.

Tuesday, March 10, 2015 at 7 p.m. GWT - London, England

If you find yourself in London, be sure to stop by the CoreOS London meetup. They are currently confirming speakers for the event. If you are interested in speaking be sure to submit on github.

Monday, March 16, 2015 at 7 p.m. PDT – San Francisco, CA

Join the CoreOS San Francisco March Meetup at Imgur (@imgur). On the agenda: Rocket, appc spec and etcd. Chris Winslett from Compose.io (@composeio) will also explain how Compose.io uses etcd as its “repository of truth.”

Friday, March 27, 2015 at 6:45 p.m. CDT – Pflugerville, TX

Brian “Redbeard” Harrington (@brianredbeard), is an opening speaker at Container Days Austin. Container Days provides a forum for all interested in the technical, process and production ramifications of adopting container style virtualization. Get your tickets here.

On another note, this month CoreOS has joined the AirPair $100K writing competition. For more details please see the contest site, https://www.airpair.com/100k-writing-competition, for more information.

If you have a CoreOS, etcd or Rocket implementation, tutorial or use case now is a great time to share. How do you apply CoreOS for automatic server updates? Do you have any stories about your implementation of etcd to keep an application up when a server needs or goes down? Any exciting ways you have applied Rocket, the first container runtime based on the Application Container specification?

If you are interested in writing about your experiences with CoreOS, etcd or Rocket, email press@coreos.com and we will give you support to make your post a success. More details about the competition are here.

App Container and Docker

2015-02-13T00:00:00+00:00

A core principle of the App Container (appc) specification is that it is open: multiple implementations of the spec should exist and be developed independently. Even though the spec is young and pre-1.0, it has already seen a number of implementations.

With this in mind, over the last few weeks we have been working on ways to make appc interoperable with the Docker v1 Image format. As we discovered, the two formats are sufficiently compatible that Docker v1 Images can easily be run alongside appc images (ACIs). Today we want to describe two different demonstrations of this interoperability, and start a conversation about closer integration between the Docker and appc communities.

rkt Running Docker Images

rkt is an App Container implementation that fully implements the current state of the spec. This means it can download, verify and run App Container Images (ACIs). And now along with ACI support the latest release of rkt, v0.3.2, can download and run container images directly from the Docker Hub or any other Docker Registry:

$ rkt --insecure-skip-verify run docker://redis docker://tenstartups/redis-commander
rkt: fetching image from docker://redis
rkt: warning: signature verification has been disabled
Downloading layer: 511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158
…
      _.-``    `.  `_.  ''-._           Redis 2.8.19 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in stand alone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 3
...
[3] 12 Feb 09:09:19.071 # Server started, Redis version 2.8.19
# redis will be  running on 127.0.0.1:6379 and redis-commander on 127.0.0.1:8081

Docker Running App Container Images

At the same time as adding Docker support to rkt, we have also opened a pull-request that enables Docker to run appc images (ACIs). This is a simple functional PR that includes many of the essential features of the image spec. Docker API operations such as image list, run image by appc image ID and more work as expected and integrate with the native Docker experience. As a simple example, downloading and running an etcd ACI works seamlessly with the addition of this patchset:

$ docker pull --format aci coreos.com/etcd:v2.0.0
$ docker run --format aci coreos.com/etcd
2015/02/12 11:21:05 no data-dir provided, using default data-dir ./default.etcd
2015/02/12 11:21:05 etcd: listening for peers on http://localhost:2380
2015/02/12 11:21:05 etcd: listening for peers on http://localhost:7001

For more details, check out the PR itself.

Docker and App Container: Looking forward

We think App Container represents the next logical iteration in what a container image format, runtime engine, and discovery protocol should look like. App Container is young but we want to continue to get wider community feedback and see the spec evolve into something that can work for a number of runtimes.

Before appc spec reaches 1.0 (stable) status, we would like feedback from the Docker community on what might need to be modified in the spec in order for it to be supported natively in Docker. To gather feedback and start the discussion, we have put up a proposal to add appc support to Docker.

We are looking forward to getting additional feedback from the Docker community on this proposal. Working together, we can create a better appc spec for everyone to use, and over time, work towards a shared standard.

Join us on a mission to create a secure, composable, and standards-based container runtime. If you are interested in hacking on rkt or App Container we encourage you to get involved:

rkt

App Container

If you want more background on the appc spec, we encourage you to read our first blog post about the App Container spec and Rocket. Also read more in a recent Q&A with OpenSource.com.

Announcing rkt and App Container v0.3.1

2015-02-06T00:00:00+00:00

Today we're announcing the next release of Rocket and the App Container (appc) spec, v0.3.1.

rkt Updates

This release of rkt includes new user-facing features and some important changes under the hood which further make progress towards our goals of security and composability.

First, the rkt CLI has a couple of new commands:

rkt trust can be used to easily add keys to the public keystore for ACI signatures (introduced in the previous release). This supports retrieving public keys directly from a URL or using discovery to locate public keys - a simple example of the latter is rkt trust --prefix coreos.com/etcd. See the commit for other examples.
rkt list is a simple tool to list the containers on the system. It leverages the same file-based locking as rkt status and rkt gc to ensure safety during concurrent invocations of rkt.

As mentioned, v0.3.1 includes two significant changes to how rkt is built internally.

Instead of embedding the (default) stage1 using go-bindata, rkt now consumes a stage1 in the form of an actual ACI, containing a rootfs and stage1 init/enter binaries, via the --stage1-image flag. This makes it much more straightforward to use alternative stage1 image with rkt and facilitates packaging for other distributions like Fedora.
rkt now vendors a copy of appc/spec instead of depending on HEAD. This means that rkt can be built in a self-contained and reproducible way and that master will no longer break in response to changes to the spec. It also makes explicit the specific version of the spec against which particular release of rkt is compiled.

As a consequence of these two changes, it is now possible to use the standard Go workflow to build the rkt CLI (e.g. go get github.com/coreos/rocket/rkt). Note however that this does not implicitly build a stage1, so that will still need to be done using the included ./build script, or some other way for those desiring to use a different stage1.

App Container Updates

This week saw a number of interesting projects emerge that implement the App Container Spec. Please note, all of these are very early and actively seeking more contributors.

Nose Cone, an independent App Container Runtime

Nose Cone is an appc runtime that is built on top of the libappc C++ library that was released a few weeks ago. This project is only a few days old but you can find it up on GitHub. It makes no use of rkt, but implements the App Container specification. It is great to see this level of experimentation around the appc spec: having multiple, alternative runtimes with different goals is an important part of building a robust specification.

Tools for building ACIs

A few tools have emerged since last week for building App Container Images. All of these are very early and could use your contributions to help get them production ready.

docker2aci

A Dockerfile and the "docker build" command is a very convenient way to build an image, and many people already have existing infrastructure and pipelines around Docker images. To take advantage of this, the docker2aci tool and library takes an existing Docker image and generates an equivalent ACI. This means the container can now be run in any implementation of the appc spec.

$ docker2aci quay.io/lafolle/redis
Downloading layer: 511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158
...
Generated ACI(s):
lafolle-redis-latest.aci
$ rkt run lafolle-redis-latest.aci
[3] 04 Feb 03:56:31.186 # Server started, Redis version 2.8.8

goaci

While a Dockerfile is a very convenient way to build, it should not be the only way to create a container image. With the new experimental goaci tool, it is possible to build a minimal golang ACI without the need of any additional build environment. Example:

$ goaci github.com/coreos/etcd
Wrote etcd.aci
$ actool -debug validate etcd.aci
etcd.aci: valid app container image

Quay support

Finally, we have added experimental support for App Container Images to Quay.io, our hosted container registry. Test it out by pulling any public image using rkt:

$ rkt trust --prefix quay.io
Prefix: "quay.io"
Key: "https://quay.io/aci-signing-key"
GPG key fingerprint is: BFF3 13CD AA56 0B16 A898  7B8F 72AB F5F6 799D 33BC
    Quay.io ACI Converter (ACI conversion signing key) <support@quay.io>
Are you sure you want to trust this key (yes/no)? yes
$ rkt run quay.io/philips/golang-outyet
$ curl 127.0.0.1:8080

While these tools are very young, they are an important milestone towards our goals with appc. We are on a path to being able to create images with multiple, independent tools (from Docker conversion to native language tools), and have multiple ways to run them (with runtimes like rkt and Nose Cone). This is just the beginning, but a great early example of the power of open standards.

Join us on a mission to create a secure, composable, and standards-based container runtime. If you are interested in hacking on rkt or App Container we encourage you to get involved:

rkt Help Wanted, Mailing list
App Container Help Wanted, Mailing list

There is still much to do - onward!

Upcoming CoreOS Events in February

2015-02-03T00:00:00+00:00

We’ve just come from FOSDEM ‘15 in Belgium and have an exciting rest of the month planned. We’ll be in Europe and the United States in February, and you can even catch Alex Polvi, CEO of CoreOS, keynoting at two events – TurboFest West (February 13) and Linux Collab Summit (February 18). Read more to see where we’ll be and meet us.

Also, thank you to all that hosted and attended our events last month. Questions or comments, contact us at press@coreos.com or tweet to us @CoreOSlinux.

Europe

See slides from the Config Management Camp 2015 talk by Kelsey Hightower (@kelseyhightower), developer advocate at CoreOS. He presented in Belgium on February 2 about Managing Containers at Scale with CoreOS and Kubernetes.

Tuesday, February 3 at 7 p.m. CET – Munich, Germany

Learn about CoreOS and Rocket at the Munich CoreOS meetup led by Brian Harrington/Redbeard (@brianredbeard), principal architect at CoreOS, and Jonathan Boulle (@baronboulle), senior engineer at CoreOS.

Tuesday, February 3 at 7 p.m. GMT – London, United Kingdom

See the first Kubernetes London meetup with Craig Box, solutions engineer for Google Cloud Platform, and Kelsey Hightower (@kelseyhightower), developer advocate at CoreOS. Attendees will be guided through the first steps with Kubernetes and Kelsey will discuss managing containers at scale with CoreOS and Kubernetes.

Thursday, February 5 at 7:00 p.m. CET – Frankfurt, Germany

Check out the DevOps Frankfurt meetup, where we will give a rundown on CoreOS and Rocket from Redbeard (@brianredbeard), principal architect at CoreOS, and Jonathan Boulle (@baronboulle), senior engineer at CoreOS.

Monday, February 9 at 7:00 p.m. CET – Berlin, Germany

Meet Jonathan Boulle (@baronboulle), senior engineer at CoreOS, at the CoreOS Berlin meetup to learn about Rocket and the App Container spec.

United States

Wednesday, February 4 at 6:00 p.m. – New York, New York

Come to our February CoreOS New York City meetup at Work-Bench, 110 Fifth Avenue on the 5th floor, where our team will discuss our new container runtime, Rocket, as well as Quay.io new features. In addition, Nathan Smith, head of engineering at Wink, www.wink.com, will walk us through how they are using CoreOS.

Monday, February 9 at 6:30 p.m. EST – New York, New York

The CTO School meetup will host an evening on Docker and the Linux container ecosystem. See Jake Moshenko (@JacobMoshenko), product manager at CoreOS, and Borja Burgos-Galindo, CEO & co-founder of Tutum, for an intro to containers and an overview on the ecosystem, followed by a presentation from Tom Leach and Travis Thieman of Gamechanger.

Friday, February 13 – San Francisco, California

See Alex Polvi, CEO of CoreOS, keynote at TurboFest West, a program of cloud and virtualization thought leadership discussions hosted by VMTurbo. Register for more details.

Tuesday, February 17 at 5:30 p.m. CST – Kansas City, Missouri

Redbeard (@brianredbeard), principal architect at CoreOS, will be kickin’ it with the Cloud KC meetup to go over CoreOS and Rocket. Thanks to C2FO for hosting this event.

Wednesday, February 18 at 10:00 a.m. PST – Santa Rosa, California

Alex Polvi, CEO of CoreOS, will present a keynote on Containers and the Changing Server Landscape at the Linux Collab Summit. See more about what Alex will discuss in a Q&A with Linux.com and tweet to us to meet at the event if you’ll be there.

Thursday, February 19 at 7:00 p.m. CST – Carrollton, Texas

Come to the Linux Containers & Virtualization meetup to meet Redbeard (@brianredbeard), principal architect at CoreOS, and learn about Rocket and the App Container spec.

February 19-February 22 – Los Angeles, California

Meet Jonathan Boulle (@baronboulle), senior engineer at CoreOS, at the SCALE 13x, the SoCal Linux Expo. Jon will present a session on Rocket and the App Container spec on Saturday, February 21 at 3:00 p.m. PT in the Carmel room.

More events will be added, so check back for updates here and at our community page!

In case you missed it, watch a webinar with Kelsey Hightower, developer advocate at CoreOS, and Matt Williams, DevOps evangelist at Datadog on Managing CoreOS Container Performance for Production Workloads

etcd 2.0 Release - First Major Stable Release

2015-01-28T08:21:00+00:00

Today etcd hit v2.0.0, our first major stable release. Since the release-candidate, in mid-December, the team has been hard at work stabilizing the release. You can find the new binaries on GitHub.

For a quick overview, etcd is an open source, distributed, consistent key-value store for shared configuration, service discovery, and scheduler coordination. By using etcd, applications can ensure that even in the face of individual servers failing, the application will continue to work. etcd is a core component of CoreOS software that facilitates safe automatic updates, coordinating work being scheduled to hosts, and setting up overlay networking for containers.

New Updates

The etcd team has been hard at work to improve the ease-of-use and stability of the project. Some of the highlights compared to the last official release, etcd 0.4.6, include

Internal etcd protocol improvements to guard against accidental misconfiguration
etcdctl backup was added to make recovering from cluster failure easier
etcdctl member list/add/remove commands for easily managing a cluster
On-disk datastore safety improvements with CRC checksums and append-only behavior
An improved Raft consensus implementation already used in other projects like CockroachDB
More rigorous and faster running tests of the underlying Raft implementation, covering all state machine and cases explained in the original Raft white paper in 1.5 seconds
Additional administrator focused documentation explaining common scenarios
Official IANA assigned ports for etcd TCP 2379/2380

The major goal has been to make etcd more usable and stable with all of these changes. Over the hundreds of pull requests merged to make this release, many other improvements and bug fixes have been made. Thank you to the 150 contributors who have helped etcd get where it is today and provided those bug fixes, pull requests and more.

Who uses etcd?

Many projects use etcd - Google’s Kubernetes, Pivotal’s Cloud Foundry, Mailgun and now Apache Mesos and Mesosphere DCOS too. In addition to these projects, there are more than 500 projects on GitHub, using etcd. The feedback from these application developers continues to be an important part of the development cycle; thank you for being involved.

Direct quotes from people using etcd:

"We evaluated a number of persistent stores, yet etcd’s HTTP API and strong Go client support was the best fit for Cloud Foundry," said Onsi Fakhouri, engineering manager at Pivotal. "Anyone currently running a recent version of Cloud Foundry is running etcd. We are big fans of etcd and are excited to see the rapid progress behind the key-value store."

"etcd is an important part of configuration management and service discovery in our infrastructure," said Sasha Klizhentas, lead engineer at Mailgun. "Our services use etcd for dynamic load-balancing, leader election and canary deployment patterns. etcd’s simple HTTP API helps make our infrastructure reliable and distributed."

"Shared configuration and shared state are two very tricky domains for distributed systems developers as services no longer run on one machine but are coordinated across an entire datacenter," said Benjamin Hindman, chief architect at Mesosphere and chair of Apache Mesos. "Apache Mesos and Mesosphere’s Datacenter Operating System (DCOS) will soon have a standard plugin to support etcd. Users and customers have asked for etcd support, and we’re delivering it as an option."

Get Involved and Get Started

After nearly two years of diligent work, we are eager to hear your continued feedback on etcd. We will continue to work to make etcd a fundamental building block for Google-like infrastructure that users can take off the shelf, build upon and rely on.

Getting started with etcd.
etcd on GitHub
Follow us on Twitter
Attend upcoming CoreOS meetups in San Francisco
Attend upcoming CoreOS meetups in New York city
Find other meetups from the CoreOS community

Brandon Philips speaking about etcd 2.0

CoreOS CTO Brandon Philips speaking about etcd 2.0 at the CoreOS San Francsico meet up:

Update on CVE-2015-0235, GHOST

2015-01-28T08:01:50+00:00

The glibc vulnerability, CVE-2015-0235, known as “GHOST”, has been patched on CoreOS. If automatic updates are enabled (default configuration), your server should already be patched.

If automatic updates are disabled, you can force an update by running update_engine_client -check_for_update.

Currently, the auto-update mechanism only applies to the base CoreOS, not inside your containers. If your container was built from an older ubuntu base, for example, you’ll need to update the container and get the patch from ubuntu.

If you have any questions or concerns, please join us in IRC freenode/#coreos.

rkt and App Container 0.2.0 Release

2015-01-23T00:00:00+00:00

This week both rkt and the App Container (appc) spec have reached 0.2.0. Since our launch of the projects in December, both have been moving very quickly with a healthy community emerging. rkt now has cryptographic signing by default and a community is emerging around independent implementations of the appc spec. Read on for details on the updates.

rkt 0.2.0

Development on rkt has continued rapidly over the past few weeks, and today we are releasing v0.2.0. This important milestone release brings a lot of new features and improvements that enable securely verified image retrieval and tools for container introspection and lifecycle management.

Notably, this release introduces several important new subcommands:

rkt enter, to enter the namespace of an app within a container
rkt status, to check the status of a container and applications within it
rkt gc, to garbage collect old containers no longer in use

In keeping with rkt's goals of being simple and composable, we've taken care to implement these lifecycle-related subcommands without introducing additional daemons or databases. rkt achieves this by taking advantage of existing file-system and kernel semantics like advisory file-locking, atomic renames, and implicit closing (and unlocking) of open files at process exit.

v0.2.0 also marks the arrival of automatic signature validation: when retrieving an image during rkt fetch or rkt run, Rocket will verify its signature by default. Kelsey Hightower has written up an overview guide explaining this functionality. This signature verification is backed by a flexible system for storing public keys, which will soon be even easier to use with a new rkt trust subcommand. This is a small but important step towards our goal of rkt being as secure as possible by default.

Here's an example of the key validation in action when retrieving the latest etcd release (in this case the CoreOS ACI signing key has previously been trusted using the process above):

$ rkt fetch coreos.com/etcd:v2.0.0-rc.1
rkt: searching for app image coreos.com/etcd:v2.0.0-rc.1
rkt: fetching image from https://github.com/coreos/etcd/releases/download/v2.0.0-rc.1/etcd-v2.0.0-rc.1-linux-amd64.aci
Downloading aci: [=============================                ] 2.31 MB/3.58 MB
Downloading signature from https://github.com/coreos/etcd/releases/download/v2.0.0-rc.1/etcd-v2.0.0-rc.1-linux-amd64.sig
rkt: signature verified: 
  CoreOS ACI Builder <release@coreos.com>

App Container 0.2.0

The appc spec continues to evolve but is now stabilizing. Some of the major changes are highlighted in the announcement email that went out earlier this week.

This last week has also seen the emergence of two different implementations of the spec: jetpack (a FreeBSD/Jails-based executor) and libappc (a C++ library for working with app containers). The authors of both projects have provided extremely helpful feedback and pull requests to the spec, and it is great to see these early implementations develop!

Jetpack, App Container for FreeBSD

Jetpack is an implementation of the App Container Specification for FreeBSD. It uses jails as an isolation mechanism, and ZFS for layered storage. Jetpack is a great test of the cross platform portability of appc.

libappc, C++ library for App Container

libappc is a C++ library for doing things with app containers. The goal of the library is to be a flexible toolkit: manifest parsing and creation, pluggable discovery, image creation/extraction/caching, thin-provisioned file systems, etc.

Get involved

If you are interested in contributing to any of these projects, please get involved! A great place to start is issues in the Help Wanted label on GitHub. You can also reach out with questions and feedback on the Rocket and appc mailing lists:

rkt

Help Wanted: https://github.com/coreos/rkt/labels/help%20wanted
Mailing list: rocket-dev@googlegroups.com

App Container

Help Wanted: https://github.com/appc/spec/labels/help%20wanted
Mailing list: appc-dev@googlegroups.com

In the SF Bay Area or NYC next week? Come to the meetups in each area to hear more about these changes and the future of rocket and appc. RSVP to the CoreOS NYC meetup and SF meetup to learn more.

Lastly, thank you to the community of contributors emerging around Rocket and App Container:

Alan LaMielle, Alban Crequy, Alex Polvi, Ankush Agarwal, Antoine Roy-Gobeil, azu, beadon, Brandon Philips, Brian Ketelsen, Brian Waldon, Burcu Dogan, Caleb Spare, Charles Aylward, Daniel Farrell, Dan Lipsitt, deepak1556, Derek, Emil Hessman, Eugene Yakubovich, Filippo Giunchedi, Ghislain Guiot, gprggr, Hector Fernandez, Iago López Galeiras, James Bayer, Jimmy Zelinskie, Johan Bergström, Jonathan Boulle, Josh Braegger, Kelsey Hightower, Keunwoo Lee, Krzesimir Nowak, Levi Gross, Maciej Pasternacki, Mark Kropf, Mark Lamourine, Matt Blair, Matt Boersma, Máximo Cuadros Ortiz, Meaglith Ma, PatrickJS, Pekka Enberg, Peter Bourgon, Rahul, Robo, Rob Szumski, Rohit Jnagal, sbevington, Shaun Jackman, Simone Gotti, Simon Thulbourn, virtualswede, Vito Caputo, Vivek Sekhar, Xiang Li

Meet us for our January 2015 events

2015-01-20T00:00:00+00:00

CoreOS CTO Brandon Philips speaking at Linux Conf AU

January has been packed with meetups and events across the globe. So far, we’ve been to India, Switzerland, France, England and New Zealand.

Check out a CoreOS tutorial from Brandon Philips (@brandonphilips) at Linux Conf New Zealand.

Our team has been on a fantastic tour meeting CoreOS contributors and friends around the world. A special thank you to the organizers of those meetups and to all those who came out to the meetups and made us feel at home. Come join us at the following events this month:

Tuesday, January 27 at 11 a.m. PST – Online

Join us for a webinar on Managing CoreOS Container Performance for Production Workloads. Kelsey Hightower (@kelseyhightower) from CoreOS and Matt Williams from Datadog will discuss trends in container usage and show how container performance can be monitored, especially as the container deployments grow. Register here.

Tuesday, January 27 at 6 p.m. EST – New York, NY

Come to our January New York City meetup at Work-Bench, 110 Fifth Avenue on the 5th floor, where our team will discuss our new container runtime, Rocket, as well as Quay.io new features. In addition, Nathan Smith, head of engineering at Wink, www.wink.com, will walk us through how they are using CoreOS. Register here.

Tuesday, January 27 at 6 p.m. PST – San Francisco, CA

Our January San Francisco meetup is not-to-miss! We’ll discuss news and updates on etcd, Rocket and AppC. Register here.

Thursday, January 29 at 7 p.m. CET – Barcelona, Spain

Meet Brian Harrington, better known as Redbeard (@brianredbeard), for CoreOS: An Overview, at itnig. Dedicated VMs and configuration management tools are being replaced by containerization and new service management technologies like systemd. This meetup will give an overview of CoreOS, including etcd, schedulers (mesos, kubernetes, etc.), and containers (nspawn, docker, rocket). Understand how to use these new technologies to build performant, reliable, large distributed systems. Register here.

Saturday, January 31-Sunday, February 1 – Brussels, Belgium

Our team is attending FOSDEM ’15 to connect with developers and the open source community. See our talks and meet the team at our dev booth throughout the event.

Redbeard (@brianredbeard) will discuss How CoreOS is built, modified, and updated on Saturday at 1 p.m. CET.
Jon Boulle (@baronboulle) from our engineering team will discuss all things Go at CoreOS on Sunday at 9:05 a.m. CET.
Kelsey Hightower (@kelseyhightower), developer advocate at CoreOS, will give a talk on Rocket and the App Container Spec at 11:40 a.m. CET.

A special shout out to the organizers of those meetups - Fintan Ryan, Ranganathan Balashanmugam, Muharem Hrnjadovic, Frédéric Ménez, Richard Paul, Piotr Zurek, Patrick Heneise, Benjamin Reitzammer, Sunday Ogwu, Tom Martin, Chris Kuhl and Johann Romefort.

If you are interested in hosting an event of your own or inviting someone from CoreOS to speak, reach out to us at press@coreos.com.

Quay.io New Features

2015-01-07T00:00:00+00:00

Today we round up our newest features and shine a spotlight on them. Since joining the CoreOS team, we have been working hard on features to improve the Quay.io experience. Highlights include squashed images (an experimental feature) for faster downloading, added build features for more control with build automation tools, and improved notification features for better communications within teams.

Squashed Images

Have you ever had to deploy a repository to a large number of machines, and wished for a faster way to do so?

We're happy to announce a new experimental feature which allows you to download a flattened version of a tag in your repository as a single layer. Instead of downloading each layer of a repository, one by one, you can download and load a single image of your repository. This cuts down on the network roundtrips associated with a pull, and even better, allows you to download your repository without using any bandwidth for deleted files. For example, if you have an image with a go binary, you can rm the go compiler after the build, and we’ll only serve the remaining files. Additionally, after you pull this image once, we will cache it so that subsequent pulls of the same image are even faster!

Using the squashed image is super simple: If you ask for tag foobar, then executing the download command will result in a new image named foobar.squash, which can be run the exact same way as its base tag:

> curl -L -f https://quay.io/c1/squash/username/reponame/foobar | docker load
> docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
quay.io/username/reponame              latest.squash       04acaed21a16        6 days ago          17.28 MB

There are some best practices for using this feature optimally. First, since you will not benefit from layering any more, you should only use this to pull to machines which will only pull the image once, and will not perform upgrades. Ephemeral or auto-scaling machines are prime candidates. Second, to take advantage of the caching, you should let the first pull of the squashed image complete before pulling it on subsequent nodes. If you plan to pull to tens, hundreds, or thousands of machines, it is best to prime the cache before doing so.

You can find the command to pull and load a squashed image in the main tree view for your repository. Please replace the placeholder credentials with the credentials of one of your own robots in order to run it. We have tons of tests which verify that these images are correct, but please test and verify that this feature works for you before adopting it in production.

Builds

The ability to build your repositories automatically from GitHub has been wildly popular. With great popularity comes great feature requests, such as: to only build the branches you specify, to test builds in different branches, and to have access to the .git directory that would be present in a clone. We’ve got two and a half of those features ready!

Building select branches

As of now, you can filter the branches which you want to build from GitHub. We allow you to specify any regular expression, which if it matches the branch name, will allow the build to proceed.

Manually triggering a branch

We also allow you to manually trigger a build from any of the branches in your repository.

Finding the revision

As for the .git directory, we were originally unable to provide it because we are not building clones; we’re building archives. After digging into your requests a bit more, it seems that the feature that you really need is to be able to tell which sha you’re actually building, at build time. So, in the interim, we've created a synthetic .git directory which will allow you to run the commands git rev-parse --short HEAD and git rev-parse HEAD. We hope this will allow you to continue forward with your build automation tools.

Communication and Notification Improvements

In order to best facilitate communication with and between your teams, we've also added notification types for Slack, Flowdock, and Hipchat. Using the API token style integrations provided by these various tools, you can now receive your repository notifications, such as build status and pushes, directly in your team chat provider.

Additional Updates

Lastly, we’ve made some ease-of-use updates to other parts of Quay.io, such as:

Inviting users to join teams can now be done by email address, and users must confirm that they would like to join
Ability to regenerate a robot’s credentials, in case you accidentally leak them (e.g. by running docker login without specifying quay.io)
Added support for the docker search command. Queries can be issued like so: docker search quay.io/somesearchterm

We look forward to hearing your feedback on the newest updates. Sign up for a free trial of Quay.io here. Or if you are interested in storing your docker containers behind the firewall, try CoreOS Enterprise Registry.

Announcing the etcd 2.0 Release Candidate

2014-12-18T00:00:00+00:00

We are pleased to announce the first release candidate for etcd v2.0.0. This is a big day for us: we have been working hard to deliver a new version of etcd and are looking forward to getting your eyes on this major release.

For those unfamiliar with etcd, it is an open source, distributed, consistent key-value store for shared configuration, service discovery, and scheduler coordination. etcd is a core component of CoreOS software that facilitates safe automatic updates, coordinates work being scheduled to hosts, and sets up overlay networking for containers. It is also actively used by other projects such as Kubernetes and Cloud Foundry, and mentioned by name in 500 projects on GitHub.

New in this Release

This etcd release improves the whole core of etcd, from the API handling internals to rethinking how to build a solid and practical Raft library. And we have taken feedback from all of you to greatly improve the operational experience with this release; thank you for taking the time to reach out on mailing lists, IRC, and in-person at our meetups. We hope that the new tooling in etcdctl and internal changes in etcd make it foolproof to configure and easy to debug.

At the heart of this release is a rethink of our Raft implementation. Raft enables etcd to do safe distributed compare-and-swap operations on keys and perform automatic leader election in the face of host failures. As many engineers have observed, implementing a consensus algorithm like Raft is a non-trivial task. But the etcd team has been hard at work. By leveraging all the latest knowledge and best practices in the Raft ecosystem, we’ve developed the new etcd Raft in 600 lines of Go and 2000 lines of tests, covering all cases in the state machine and every case described in the paper.

Working with the Community

We’re already hearing good things about the new implementation from our community:

“I ran across etcd's raft package and really liked it! It's such a nice minimal raft library that I could embed that in Beehive in 1 day with less than 200 LOC.”

– Soheil Hassas Yeganeh author of Beehive

And from the co-author of the Raft paper:

"I appreciate how you're being careful to preserve the semantics of the unoptimized protocol."

– Diego Ongaro co-author of the Raft paper

We also owe a hearty thank you to Blake Mizerany for his architecture guidance, and Ben Darnell and the CockroachDB team who have provided a lot of feedback on and contributions to our Raft library and recently finished integrating it into their own project.

With this new solid core, etcd will become an even better fundamental building block for composing distributed systems. To give it a shot, please head on over to our GitHub release page for container images and binaries for Linux and OS X.

Frequently Asked Questions

What happened to etcd v1.0?

Early in the development cycle of etcd we had an API that was called "v1". This API was on the path for removal since the v0.2.0 release over a year ago. Since we are removing this "v1" API we decided to make the major number line up with the API version number, thus etcd v2.0.0. Up until today v2.0.0 was the v0.5.0 development that has been happening in the master branch.

Is etcd v0.4 compatible with v2.0.0?

For clients, the v2 API is fully supported and shouldn’t require any changes to your code unless you’re relying on older metadata about the etcd nodes. The mechanism for peer communication has been improved through the work on the Raft library and isn’t compatible with v0.4 peers. A few other implementation details were also changed and are described in the backwards compatibility document.

When will etcd v2.0.0 appear in CoreOS?

After the release of etcd v2.0.0, it will first appear in the CoreOS alpha channel and then work its way into the beta and stable channels, following the normal CoreOS release process. If you’d like to use etcd v2.0.0 on CoreOS before its official release, it can easily be run as a container.

Is etcd v2.0.0 available as a container?

Yes! The releases page on GitHub has instructions for containers, Linux, and OS X.

App Container Spec One Week In

2014-12-09T00:00:00+00:00

Last week we announced an initial version of the "App Container Spec", a proposed standard for building and running application containers, along with a prototype implementation called Rocket. The community's engagement with the specification this week has been highly productive, with people submitting numerous improvements and contributing feedback on a variety of topics ranging from crypto algorithms to DNS-based image discovery mechanisms. We had hoped to kick off a community effort to design the specification and it is already working beyond our expectations. With discussions starting on implementing the spec in Mesos, Docker, and Kubernetes, it is clear that there is an incredible level of interest in the community in developing this new container standard.

GitHub Organization

To keep the momentum going and ensure the project is a collaborative and community-driven effort, today we are pulling the App Container specs and tools out into a new GitHub organization called appc. We have also created a Google Group dedicated to discussion about the specification, appc-dev, and an IRC channel on Freenode, #appc. These changes should make it easier for you to get more actively involved or just more easily stay on top of changes to the specification. We are eagerly looking forward to bringing on more maintainers as new implementations of the spec emerge.

You can read more about the spec and its goals over on the new repository on GitHub.

Docker 1.3.2 in Stable Channel

2014-12-03T00:00:00+00:00

Update: 494.3.0 wasn't correctly applying this shim, we've addressed this in 494.4.0 which is rolling out now.

This morning we started rolling out 494.4.0 to the stable channel. Among other things, this version updates Docker to 1.3.2, following in the footsteps of our releases to the alpha and beta channels last week.

The update will introduce the Docker --insecure-registry flag and, by default, require secure connections to container registries. This flag is not backwards compatible and will prevent many users from being able to deploy containers from self-run private repositories without SSL configured.

In order to temporarily ensure backward compatibility, the Docker daemon will be run with --insecure-registry=0.0.0.0/0 by default in version 494.4.0 of CoreOS.

As a CoreOS user, this means that your insecure private registries will continue to temporarily work in the same way as before, but we plan to remove this shim and require each user to manage whether or not their machines will communicate with an insecure registry.

Before January 12th, 2015, CoreOS machines which connect to insecure registries will need to add the appropriate flags to the Docker daemon. The preferred method is via a systemd drop-in:

$ cat /etc/systemd/system/docker.service.d/50-insecure-registry.conf
[Service]
Environment=DOCKER_OPTS='--insecure-registry="10.0.1.0/24"'

This step can be automated by coreos-cloudinit by adding the following to your cloud-config:

#cloud-config

write_files:
  - path: /etc/systemd/system/docker.service.d/50-insecure-registry.conf
    content: |
        [Service]
        Environment=DOCKER_OPTS='--insecure-registry="10.0.1.0/24"'

Again, if you depend on insecure registries, you will need to update your configuration by January 12th, 2015.

As always, if you have any questions or concerns, please join us in IRC #coreos.

CoreOS is building a container runtime, rkt

2014-12-01T00:00:00+00:00

rkt is a new container runtime, designed for composability, security, and speed. Today we are releasing a prototype version on GitHub to begin gathering feedback from our community and explain why we are building rkt.

Why we are building rkt

When we started building CoreOS, we looked at all the various components available to us, re-using the best tools, and building the ones that did not exist. We believe strongly in the Unix philosophy: tools should be independently useful, but have clean integration points. We hope this is reflected in tools that we build, such as etcd, which have seen widespread adoption and use outside CoreOS itself.

When Docker was first introduced to us in early 2013, the idea of a “standard container” was striking and immediately attractive: a simple component, a composable unit, that could be used in a variety of systems. The Docker repository included a manifesto of what a standard container should be. This was a rally cry to the industry, and we quickly followed. Brandon Philips, co-founder/CTO of CoreOS, became a top Docker contributor, and now serves on the Docker governance board. CoreOS is one of the most widely used platforms for Docker containers, and ships releases to the community hours after they happen upstream. We thought Docker would become a simple unit that we can all agree on.

Unfortunately, a simple re-usable component is not how things are playing out. Docker now is building tools for launching cloud servers, systems for clustering, and a wide range of functions: building images, running images, uploading, downloading, and eventually even overlay networking, all compiled into one monolithic binary running primarily as root on your server. The standard container manifesto was removed. We should stop talking about Docker containers, and start talking about the Docker Platform. It is not becoming the simple composable building block we had envisioned.

rkt + App Container

We still believe in the original premise of containers that Docker introduced, so we are doing something about it. While we are at it, we are cleaning up and fixing a few things that we’d like to see in a production ready container. What is important to us in the design of a container?

Composable. All tools for downloading, installing, and running containers should be well integrated, but independent and composable.
Security. Isolation should be pluggable, and the crypto primitives for strong trust, image auditing and application identity should exist from day one.
Image distribution. Discovery of container images should be simple and facilitate a federated namespace, and distributed retrieval. This opens the possibility of alternative protocols, such as BitTorrent, and deployments to private environments without the requirement of a registry.
Open. The format and runtime should be well-specified and developed by a community. We want independent implementations of tools to be able to run the same container consistently.

rkt 0.1.0

rkt is a command line tool, rkt, for running App Containers. An “App Container” is the specification of an image format, container runtime, and a discovery mechanism. rkt is the first implementation of an App Container, but we do not expect it to be the only one.

App Container Image: definition of a signed/encrypted tgz that includes all the bits to run an app container.
App Container Runtime: definition of the environment the running app container should be given.
App Container Discovery: a federated protocol for finding and downloading an app container image.

The best way to drive a standard is to let a successful and proven implementation become the de facto one. However, when the point of the software is interoperability (as it is with containers), we think things are different.

In developing the App Container specification, we started with some thoughtfulness around the requirements up front, then spent time refining it as we worked on an implementation (rkt). We feel these specs and implementations are pretty well thought out, but are still early enough that we need your help refining it. Please contribute to the spec by sending a pull request to start the discussion.

App Container Image

An App Container Image (ACI) is a specification for the image format of a container. It is a simple flat tarball that is always signed and optionally encrypted. By convention, an ACI is minimal, meaning it only includes the bits absolutely required to execute the application. Since an ACI may be encrypted, distribution via systems like BitTorrent, public object storage, or mirror networks is a possibility. Think of ACI as a variant of Amazon’s AMI, but created for a container world.

Read about and contribute to the ACI draft.

App Container Runtime

The App Container Runtime defines what environment and facilities a container runtime should provide. This includes devices, environment variables, and privileges that a container should expect. It also includes a definition of a meta-data service interface for exposing data to the environment from outside the container.

Security primitives are very important to us, so we added an identity feature to the meta-data service. This means every instance of a running container is given a unique identity, coupled with a lightweight HSM-like service for signing. No existing VM or container environment has a concept like this, so we would welcome and appreciate community feedback on this design.

Read about and contribute to the runtime draft.

App Container Discovery

App Container Discovery is a method for finding your app container image. It is inspired by golang’s vanity URL convention for import paths. This means you’ll be able to refer to containers with simple names like coreos.com/etcd, allowing organizations to federate their downloads without running their own registry.

Standard Containers

We want the world to run containers. A world where your application can be packaged once, and ran in the environment you choose. We believe rkt and a definition around the App Container is a requirement for this to work. Please contribute your thoughts to our mailing list, our source code or join us in person at the CoreOS meet-up tonight in San Francisco.

FAQ

What is rkt?

rkt is an alternative to the Docker runtime, designed for server environments with the most rigorous security and production requirements. rkt is oriented around the App Container specification, a new set of simple and open specifications for a portable container format.

When is rkt available?

rkt is available today on GitHub. We are releasing a 0.1.0 prototype to gather community feedback. Please note that this is a prototype quality release, very much in the spirit of “release early, release often”. Please provide feedback via GitHub.

Why not just fork Docker?

From a security and composability perspective, the Docker process model - where everything runs through a central daemon - is fundamentally flawed. To “fix” Docker would essentially mean a rewrite of the project, while inheriting all the baggage of the existing implementation.

Why are you doing this now?

At CoreOS we have large, serious users running in enterprise environments. We cannot in good faith continue to support Docker’s broken security model without addressing these issues. Additionally, in the past few weeks Docker has demonstrated that it is on a path to include many facilities beyond basic container management, turning it into a complex platform. Our primary users have existing platforms that they want to integrate containers with. We need to fill the gap for companies that just want a way to securely and portably run a container.

Will rkt run on [Ubuntu, RHEL, CentOS, etc]?

Yes, rkt is a stand alone tool that will live outside of CoreOS itself and can be used on a variety of platforms. In a sense it is similar to the etcd project in that it is a tool we built because nothing like it existed.

What is the difference between rkt and App Container?

“App Container” defines a specification of the facilities surrounding the container. rkt implements these facilities as a command line tool. An open specification allows other systems do their own implementation of App Container without using rkt at all. CoreOS fully supports and embraces alternative implementations.

Could App Container support be contributed to the Docker Platform?

Definitely. If the App Container specifications were implemented inside of Docker, the projects will be interoperable, meeting the original goal of the manifesto. CoreOS will evaluate contributing this work once App Container matures.

Will CoreOS continue to ship Docker?

Yes. We will continue to make sure CoreOS is the best place to run Docker. We will save the details for a future post, once rkt has developed further, but expect Docker to continue to be fully integrated with CoreOS as it is today.

Docker 1.3.2 Rolled Out Today

2014-11-24T00:00:00+00:00

Today Docker released Docker 1.3.2, immediately after which we began automatically rolling out an OS update. We were able to work with Docker to ensure that the CoreOS update could ship as soon as the Docker patch was released to the public. At the time of this writing, a significant number of CoreOS instances have already been updated.

Along with the aforementioned security patches, Docker 1.3.2 introduced the --insecure-registry flag which accepts a CIDR notation block of whitelisted addresses. If your installation requires access to an insecure registry, you will need to launch the Docker daemon with the --insecure-registry flag and the CIDR block appropriate for your deployment. The recommended method is via a write_files directive in your cloud-config.

#cloud-config

write_files:
  - path: /etc/systemd/system/docker.service.d/50-insecure-registry.conf
    content: |
        [Service]
        Environment=DOCKER_OPTS='--insecure-registry="10.0.1.0/24"'

As a practice, CoreOS doesn’t like to roll out breaking updates like this, but the security content of Docker 1.3.2 was deemed important enough to make an exception.

If you have any questions or concerns, please join us in IRC #coreos.

CoreOS Brings Kubernetes to Any Cloud Platform

2014-11-10T00:00:00+00:00

If you aren’t familiar with Kubernetes, it is a container cluster manager from Google that offers a unique workflow for managing containers across multiple machines. We have written about Kubernetes and CoreOS in a tutorial series (Part 1, Part 2) here for those wanting to learn more. Last week Google announced Google Container Engine, which enables you to run and manage Docker containers on Google Cloud Platform’s virtual machines.

But for those wanting to run Kubernetes outside of Google Cloud Platform or ensure workload mobility, CoreOS provides an easy way to use Kubernetes in any cloud or on-prem environment. CoreOS maintains a full tutorial on running an elastic Kubernetes cluster on EC2. Running Kubernetes on non-Google platforms is made possible by the CoreOS project flannel, which provides cross-container networking on any cloud provider.

In addition to flannel, Kubernetes utilizes another CoreOS project, etcd, as its distributed k/v store. etcd was inspired by a paper written by Google and was developed by CoreOS because an open-source solution to this important piece of instrastructure didn't exist. It is exciting to see this effort come full circle as etcd used as a key piece of Kubernetes.

Join us in IRC or on GitHub to learn more. Or explore our documentation on how to get started with CoreOS

Weekend Enjoyment: CoreOS Deployment Videos

2014-11-07T00:00:00+00:00

For your weekend enjoyment, we’ve posted a few videos that showcase unique CoreOS deployments. First, Playstation walks through how their developers set up structured but customizable dev environments for their software teams using CoreOS, docker and systemd-nspawn.

For application testing, MemSQL runs a 107 machine CoreOS cluster. To obtain the most accurate and consistent results, this cluster must run on bare metal and CoreOS was the perfect tool for the job.

Production application deployments on CoreOS are highly extensible by design. Layer shows us how CoreOS runs the infrastructure to support their unique communications platform. Mailgun has developed an etcd-aware load balancer that handles traffic routing for their API infrastructure. Both examples show how CoreOS enables platform versatility and increases the productivity of your devops team.

We hope these videos have inspired you to start hacking on your CoreOS-powered infrastructure this weekend. Enjoy!

Announcing CoreOS Enterprise Registry, a secure Docker registry behind your firewall

2014-10-30T00:00:00+00:00

Today we are launching CoreOS Enterprise Registry, a secure Docker registry behind the firewall. The first enterprise-ready solution, CoreOS Enterprise Registry provides a web-based management interface that includes:

permission controls
auditing capabilities
deployment of containers safely and securely
availability on-prem and in your datacenter

CoreOS Enterprise Registry provides easy ways to assign and revoke credentials as needed, and even goes as granular as providing per-user permissions. For example, members of a team can be allowed to modify repos or only have docker pull ability. It also provides auditing capabilities so you can see who last made repository commits or access changes. Further advance controls include robot accounts, a special account type for driving automation tools and other circumstances where you don't need UI access.

Once just part of our Premium Managed Linux offering, CoreOS Enterprise is now available for as little as $10 a month. We invite you to try it out.

To see a demo of CoreOS Enterprise Registry and talk in-person, join us this Monday for our meetup hosted at PlayStation in San Mateo.

A Meetup Ride to San Mateo

2014-10-29T00:00:00+00:00

Our San Francisco meetup is hitting the road on Monday and going to PlayStation in San Mateo! As you may or may not know we are big fans of buses around here. So we decided to get a biodiesel and solar powered bus to take a group down to the meetup in style. If you are in San Francisco and want to ride with us, be one of the first 20 to sign up here and I’ll get back to you with details.

We only have 20 spots available so be absolutely positive you can make the meetup on Monday!

CoreOS Now Available On Microsoft Azure

2014-10-20T00:00:00+00:00

This morning at the Microsoft Azure press event, Microsoft announced that CoreOS is now available as an image in the Azure Virtual Machines Gallery.

Launching a CoreOS Linux VM is as easy as logging into the Azure Virtual Machines Gallery, and selecting the CoreOS image. This initial release of CoreOS Linux on Microsoft Azure includes the latest version of CoreOS Linux from the CoreOS alpha channel and will reach stable over the coming weeks.

In addition to everything you need for running Docker containers out of the box, Azure customers running CoreOS can now leverage CoreOS’ distributed key/value store, etcd, to manage cluster state and configuration across an entire fleet of virtual machines. Azure customers also have access to CoreOS’ automatic software updates and patches, which means never having to spend time manually rolling out OS updates again.

The CoreOS team is really excited to be a part of Microsoft’s commitment to the adoption of open source technologies.

Godep for End User Go Projects

2014-10-15T00:00:00+00:00

Every language has a problem with dependencies. In C there are problems with bumping minor shared library versions and having unexpected bugs appear in applications. Ruby and Python have problems with applications having conflicting dependencies requiring the use of bundle or virtualenv to create private library stores.

Go has a distinct advantage because it is a statically compiled language. In other words, a Go program always carries its runtime dependencies with it; no possibilities of library versions changing. However, creating a developer environment in which to compile this static binary is a bit more difficult and some tools are required.

While working on etcd, a project written in Go, we took an evolutionary path towards a dependency system that satisfied a few basic goals we had:

Reproducible builds: given the same git hash and version of the Go compiler we wanted an identical binary every time.
Zero dependencies: developers should be able to fork on GitHub, make a change, build, test and send a PR without having anything more than a working Go compiler installed.
Cross platform: compile and run on OSX, Linux and Windows. Bonus points for cross-compilation.

tl;dr We arrived at using godep but the rest of the post below has some insights on how we arrived here.

Checked in GOPATH

To satisfy the need for reproducible builds and zero dependencies we checked in a copy of the GOPATH to third_party/src. However, this revealed several problems over time:

go get github.com/coreos/etcd was broken since downstream dependencies would change master and go get would setup a GOPATH that looked different than our checked in version.
Windows developers had to have a working bash. Soon, we were maintaining a copy of our build script written in Powershell.

We felt that go get wasn't a useful tool for installing etcd since it was just a project built in Go and go get is primarily useful for easily grabbing libraries when you are hacking on something. However, we were overwhelmed every week with bug reports from users who wanted to simply go get github.com/coreos/etcd.

To solve the Windows problem I created third_party.go which ported the GOPATH management shell script to Go and gave us the ability to drop the Powershell.

third_party.go

third_party.go worked well for a few weeks and we could remove the duplicate build logic in the Powershell scripts. The basic usage was simple:

# Bump the raft dependency in the custom GOPATH
go run third_party.go bump github.com/coreos/go-etcd
# Use third_party.go to set GOPATH to third_party/src and build
go run third_party.go build github.com/coreos/etcd

But, there was a fatal flaw with this setup: it broke cross compilation via GOOS and GOARCH.

GOOS=linux go run third_party.go build github.com/coreos/etcd
fork/exec /var/folders/nq/jrsys0j926z9q3cjp1yfbhqr0000gn/T/go-build584136562/command-line-arguments/_obj/exe/third_party: exec format error

The reason is that GOOS and GOARCH get used internally by go run. Meaning it literally tries to build third_party.go as a Linux binary and runs it. Running a Linux binary on an OSX machine doesn't work.

Furthermore by using third_party.go didn't get us any closer to being "go gettable". With the continued weekly emails and bugs for this I started looking around for better solutions and found goven.

goven and goven-bump

goven achieves all of the desirable traits: reproducible builds, zero dependencies to start developing, cross compilation, and as a bonus go install github.com/coreos/etcd works!

The basic theory of operation is it checks all dependencies into subpackages of your project. Instead of importing code.google.com/p/goprotobuf you import github.com/coreos/etcd/third_party/code.google.com/p/goprotobuf. It makes the imports uglier but it is automated by goven.

Along the way I wrote some helper tools to assist in bumping dependencies which can be found on GitHub at philips/goven-bump. The scripts goven-bump and goven-bump-commit grab the hg revision or git hash of the dependency along with running goven. This makes bumping a dependency and getting a basic commit message as easy as:

cd ${GOPATH}/github.com/coreos/etcd
goven-bump-commit code.google.com/p/goprotobuf
git commit -m 'bump(code.google.com/p/goprotobuf): 074202958b0a25b4d1e194fb8defe5d69c300774'

godep save

The way that goven re-wrote the package paths turned out to be a very successful model for us and saved the developers of etcd a ton of headaches. However, the goven-bump scripts were rather annoying and it forced the developer to use git log to find the hash of the upstream dependency.

After emailing and chatting with Keith Rarick, author of goven and godep, he told me that he was planning on adding a feature to his godep project that rewrote import paths just like goven but had the advantages of a static dependency manifest file. After a few months of nagging he merged the godep save -r feature and I could finally start converting projects to stop using goven-bump. Woo!

godep adds some additional complexity for the etcd team. But, the simplicity it presents to regular contributors and users used to go get make it worth the additional effort. For all of the CoreOS non-library projects that are written in Go we now use godep and have achieved all of our goals:

Reproducible builds
Zero dependencies
Cross platform
Easy to introspect

If you maintain a Go project, I highly recommend you go install github.com/tools/godep.

Managing CoreOS with Ansible

2014-10-13T00:00:00+00:00

This is a guest post by Roman Shtylman

This post will cover basic techniques for managing CoreOS machines using Ansible. Familiarity with Ansible and basic understanding of CoreOS are helpful in following along with this post.

What is Ansible?

From the Ansible Documetation

Ansible is an IT automation tool. It can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments or zero downtime rolling updates.

At the most basic level, Ansible is a tool that will run sets of commands (typically over SSH) on remote boxes (called the inventory). These commands can be as simple as one line shell statements, or use any of the built-in Ansible modules for common tasks like file copying, package management, system information, and more. Ansible ships with many useful modules and you can easily create your own.

Why Ansible for CoreOS?

Ansible does not require a remote agent running on the target machine. It can perform all of its functions over a basic SSH connection.

CoreOS is a minimal Linux distribution meant for running containers. It does not ship with a package manager or many of the other common system elements one might expect coming from other more desktop oriented distributions.

Because Ansible does not require a remote agent, and because CoreOS design favors running containers over system software directly on the machine, the two make a good fit.

Getting Started

Before continuing, make sure that you have Ansible installed on your local machine. If Ansible is properly installed, you should be able to run the following command in a shell.

$ ansible --version
ansible 1.8

We have prepared a sample repository with a Vagrant file to demonstrate using Ansible with CoreOS locally. This is also a good way to test your playbooks (sets of Ansible commands) to make sure they are working as you expect. You will need to install vagrant to use these samples.

If you are already running Ansible and are familiar with launching CoreOS machines in existing cloud providers, you can ignore these steps and jump to the next section

Run the following commands from your local machine:

$ git clone https://github.com/defunctzombie/coreos-ansible-example.git
$ cd coreos-ansible-example
$ vagrant up
-- wait for vagrant to finish booting the machine(s) --
$ ./bin/generate_ssh_config

This will boot a CoreOS machine and configure it with some basic networking. After the machine has booted, we will run a local script generate_ssh_config to create a configuration file for Ansible so that it knows how to access our machine over SSH.

Inventory Setup

The inventory file defines the hosts and groups. Our vagrant example has an inventory file in inventory/vagrant which we will use when configuring our example CoreOS hosts with ansible.

## inventory file for vagrant machines
core-01 ansible_ssh_host=172.12.8.101

[web]
core-01

We only have 1 host called core-01 and we have created a web group and listed our host under this group. You can have any number of hosts and groups. Ansible even support dynamic inventory files which is what you are likely to use in large scale production environments.

To run a test ping against our vagrant inventory, execute the following command in your shell (from the project folder).

$ ansible -i inventory/vagrant all -m setup

If everything worked as expected you should see the following output.

core-01 | FAILED >> {
    "failed": true,
    "msg": "/bin/sh: /usr/bin/python: No such file or directory\r\n",
    "parsed": false
}

The command has failed. To understand why this happened and how to fix it, let's take a closer look at how Ansible runs commands on remote machines.

Getting Ansible Running

When you run Ansible commands or playbooks, Ansible will ssh into the remote machine, copy over the module code (written in Python) and run the module using the arguments specified in the playbook.

The target machine must have a Python interpreter for Ansible to be able to execute these modules and thus configure your machine.

CoreOS is designed for running containers and does not ship with a Python interpreter. Additionally, it has no package manager to install Python. This presents a small chicken-and-egg problem.

Luckily Ansible has a raw execution mode which bypasses Python modules and runs shell commands directly on a remote system. We will leverage this feature to bootstrap a lightweight Python interpreter onto our CoreOS hosts. Once the hosts are bootstrapped with Python, playbooks can leverage the myriad of provided Ansible modules to perform system tasks like start services, install Python libraries, and manage Docker containers.

Edit the inventory/vagrant file and add the following items at the end

[coreos]
core-01

[coreos:vars]
ansible_ssh_user=core
ansible_python_interpreter="PATH=/home/core/bin:$PATH python"

This will configure Ansible to look for python and pip in /home/core/bin and use it for all hosts in the coreos group. Without this, Ansible will try to use /usr/bin/python which does not exist on our CoreOS hosts.

To bootstrap our CoreOS hosts we will use the coreos-bootstrap role.

Install the role using the following command

$ ansible-galaxy install defunctzombie.coreos-bootstrap -p ./roles

Now we can run the provided bootstrap.yml file using ansible.

$ ansible-playbook -i inventory/vagrant bootstrap.yml

Once this command has completed, we can run our original ansible setup command and see a list of the gathered facts.

$ ansible -i inventory/vagrant all -m setup
core-01 | success >> {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "172.17.42.1",
            "10.0.2.15",
            "172.12.8.101"
        ],
    ...

Take a moment to look at bootstrap.yml and site.yml. Notice that bootstrap.yml is included first. Your own Ansible scripts will need to similaly have a bootstrap.yml or playbook which configures CoreOS hosts before running other playbooks.

Example Playbook

Once Ansible can run successfully on your CoreOS hosts, we can do things like start system services or launch Docker containers.

The website.yml file shows an small example. It starts the etcd service. Then it installs the docker-py library using pip and finally uses the Ansible docker module to launch a container on the web host group.

- name: example nginx website
  hosts: web
  sudo: true
  tasks:
    - name: Start etcd
      service: name=etcd.service state=started

    - name: Install docker-py
      pip: name=docker-py

    - name: pull container
      raw: docker pull nginx:1.7.1

    - name: launch nginx container
      docker:
        image="nginx:1.7.1"
        name="example-nginx"
        ports="8080:80"
        state=running

Run this playbook with the following command

$ ansible-playbook -i inventory/vagrant website.yml

You can now open http://172.12.8.101:8080 and see the default nginx landing page.

You are now ready to create more plays to configure your CoreOS hosts. Plays can be leveraged for many tasks like automated app deployment, cluster management, and more.

Tips

Local Docker Registry

Downloading imagines from remote registries can be time and bandwidth consuming. You can speed up deployments to a set of machines by running a local registry for a cluster and having other machines pull from the registry. This can be easily automated with roles and playbooks.

Install pip modules in a common playbook

If you will be using the docker module, consider installing docker-py via the pip module in a playbook that runs on all hosts and comes after the bootstrap.yml playbook.

Prefer containers over local binaries

Avoid over-configuring the CoreOS hosts with too many locally installed tools and tweaks. In many cases, containerized services will do the job just as well and can be granted limited access to the underlying host.

You can even monitor host processes by bind mounting the relevant paths into a container.

CoreOS Machines Secured from Shellshock

2014-09-26T00:00:00+00:00

Over the last few hours we began rolling out shellshock fixes to all CoreOS deploys that have automatic updates enabled (this is default). If you have disabled automatic upgrades through cloud-config reboot-strategy, a manual reboot should give you the latest version.

You can test if your CoreOS has been patched by testing the exploit, or by checking the version of CoreOS you are running in /etc/os-release. Patched versions are listed below:

Channel	Version	Release Notes
Alpha	452.0.0	View Release Notes
Beta	444.2.0	View Release Notes
Stable	410.1.0	View Release Notes

This is an important example of our update philosophy. We believe the key to security is not just making patches available, but also applying them. Everything we build is to make this process safe. Hours after the real fix was published, we built new images, deployed them to every CoreOS platform, and patched the CoreOS slice of the internet.

Only half the problem

An astute reader might ask: what happens to the bash instances inside Docker containers? This is indeed a problem. One of the issues with building an image for every application is that you have sprawl: now there are multiple versions of bash, running in the various Docker containers.

Once Docker updates their base images, you will need to rebuild every image in your environment, as well as re-deploy it.

Updating containers is an open problem, and we intend to help with this by applying our update model to containers as well. More to come on this soon, but we hope this model will make huge improvements in the security of the internet.

Security Update on CVE-2014-6371 Shellshock

2014-09-25T00:00:00+00:00

As many of you have heard, there are open Bash vulnerabilities, CVE-2014-6271, and CVE-2014-7169. The common vectors for arbitrary code execution from these CVEs include: bash exposed via certain web applications and dhcpcd scripts. Additionally, SSH accounts restricted by the command= option in their ssh key can bypass that restriction.

By default, CoreOS is not configured in a way that would allow these issues to be exploited. Regardless, we are working to further protect our users by incorporating the fixes, once released and verified, into a new build. We were prepared to roll out a new release yesterday (September 24th, 2014), but the original fix for the CVEs proved to be insufficient by the developers.

If you are running containers on top of CoreOS that utilize a vulnerable bash and that bash is exposed to the open internet via a mechanism like CGI, then you will need to update your containers accordingly.

Versions of CoreOS with vulnerable bash

All CoreOS channels including, Stable, Beta, and Alpha ship with bash 4.2.20 which is affected by CVE-2014-6271 and CVE-2014-7169.

$ bash --version
GNU bash, version 4.2.20(1)-release (x86_64-cros-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

When will this be fixed?

CoreOS is actively tracking the open CVEs and is prepared to release 410.1.0 (Stable), 440.2.0 (Beta), and an Alpha release once CVE-2014-6371 and CVE-2014-7169 have been patched upstream and verified by the CoreOS team.

What can I do to protect myself?

Disable the use of dhcpcd on untrusted networks. dhcpcd is not the default DHCP mechanism on CoreOS.
Disable the use of the SSH option AcceptEnv in sshd.conf and ssh keys restricted by a command option in authorized_keys.
Update any containers which expose Bash to the open internet via a mechanism like CGI
If you have disabled automatic application of updates be sure to monitor your systems for new updates and reboot your machines accordingly.

Congrats to Interactive Markdown at the TechCrunch Disrupt Hackathon

2014-09-08T00:00:00+00:00

Yesterday we were excited to see that Interactive Markdown, who used CoreOS on DigitalOcean, won runner up at the TechCrunch Disrupt hackathon. CoreOS was made available on DigitalOcean on Friday and come Saturday the Interactive Markdown team was building their application on top of CoreOS. From the TechCrunch Disrupt submission:

We set up a sophisticated CoreOS cluster on DigitalOcean to support live streaming of Android mobile applications via a highly optimized VNC dynamic reverse proxy directly to a web canvas. This streaming works on both desktop and mobile devices. We also set up a build process in docker to allow the running of numerous languages on a generic backend platform written in Go.

In just 24 hours the team had a working web application of Interactive Markdown, all built on CoreOS on DigitalOcean. Pretty impressive to get that going and present to an audience of 1,000 or so hackers in 24 hours. Congrats again to the team and go check out their TechCrunch Disrupt Hackathon demo here:

If you're interested in running your own CoreOS cluster on DigitalOcean, follow our step by step guide.

CoreOS Image Now Available On DigitalOcean

2014-09-05T00:00:00+00:00

2,000 of you voted for CoreOS on DigitalOcean and we're happy to report that it's arrived. Starting today, you can head to DigitalOcean, choose the CoreOS image, and start experimenting with containers and all CoreOS has to offer.

In fact, if you want to try out CoreOS on DigitalOcean, use this limited edition promo code, SFDOCOREOS25 (expires 9/12/2014) and receive a $25 credit to your DigitalOcean account.

The step by step guide can get you started with a single droplet.

CoreOS Channels

CoreOS is released into alpha, beta, and stable channels for each provider we support. Today, we’re releasing the first CoreOS alpha image for DigitalOcean at version 429.0.0. The first beta image will be available on DigitalOcean when an alpha image is upgraded to beta, which happens roughly every two weeks. The stable image will follow the same pattern.

If you’re interested in booting the alpha image today but would like to track the stable or beta channel for all future updates, you can switch your release channel very easily after boot.

Get Started with CoreOS on DigitalOcean

CoreOS is designed to run in clusters and is optimized for running containers. The init system for your cluster is fleet, a cluster-aware tool built on top of systemd. If you’re unfamiliar with systemd, it’s the next-generation init system being adopted by all of the major Linux distributions -- Ubuntu, Fedora, Debian, Red Hat, and more.

Here’s a collection of guides that will help you get started:

Introducing flannel: An etcd backed overlay network for containers

2014-08-28T00:00:00+00:00

Edit: This post has been updated to reflect the project name change from rudder to flannel

As we have previously blogged, Kubernetes, the container cluster manager, works great with CoreOS to distribute a workload across your entire cluster. To make it easier to find your services, Kubernetes does away with port-mapping and assigns a unique IP address to each pod. This works well on Google Compute where each host is assigned a /24 for use by individual pods. Things are not as easy on other cloud providers where a host cannot get an entire subnet to itself. flannel aims to solve this problem by creating an overlay mesh network that provisions a subnet to each server.

While flannel was originally designed for Kubernetes, it is a generic overlay network that can be used as a simple alternative to existing software defined networking solutions.

How it works

An overlay network is first configured with an IP range and the size of the subnet for each host. For example, one could configure the overlay to use 10.100.0.0/16 and each host to receive a /24 subnet. Host A could then receive 10.100.5.0/24 and host B could get 10.100.18.0/24. flannel uses etcd to maintain a mapping between allocated subnets and real host IP addresses. For the data path, flannel uses UDP to encapsulate IP datagrams to transmit them to the remote host. We chose UDP as the transport protocol for its ease of passing through firewalls. For example, AWS Classic cannot be configured to pass IPoIP or GRE traffic as its security groups only support TCP/UDP/ICMP.

Getting flannel

Download and build flannel from GitHub: https://github.com/coreos/flannel.

Technical details

flannel uses etcd for storage of both configuration data and subnet assignments. Upon startup, a flannel daemon will retrieve the configuration and a list of subnet already in use. It will select an available subnet (a randomly picked one) and attempt to register it by creating a key in etcd. For example, consider the following configuration:

Overlay network range: 10.100.0.0/16
Size of subnet for each host: /24

and current registrations stored as keys under /coreos.com/network/subnets:

10.100.5.0-24
10.100.13.0-24
10.100.17.0-24
10.100.18.0-24

flannel might pick 10.100.15.0/24 and format the key as 10.100.15.0-24. It will then attempt to create this key. If it succeeds, it has acquired a subnet lease for 24 hours. flannel uses etcd TTL on keys to enforce lease expirations. Should the key creation fail, it indicates that another host managed to acquire the same subnet first and flannel will retry the registration procedure. An hour before its lease expiration, flannel will extend its lease by doing an update.

The value stored for each key contains the real IP of the host. flannel keeps an eye (via etcd directory watch) on all entries stored in /coreos.com/network/subnets and uses this information to maintain a routing table. To perform the encapsulation, flannel utilizes Universal TAP/TUN devices (in TUN mode) and proxies the IP fragments between the TUN device and a UDP socket.

Performance

To find out how much performance penalty flannel introduces, we ran both latency and bandwidth tests on AWS m3.medium VM. We used qperf tool to gather the measurements:

	Without flannel	With flannel
UDP Latency	133 us	201 us
TCP Bandwidth	47.8 MB/sec	47.2 MB/sec

As can be seen from the results, while flannel introduces non-trivial latency penalty, it has almost no affect on the bandwidth.

Future direction

flannel is still in the early phases of development and should be considered to be in the experimental stage. Apart from stabilizing current functionality, future plans include supporting additional encapsulations such as IPSEC. Moreover, even for those environments where it is feasible for a host to be assigned a routable subnet, we would like to extend flannel to perform the subnet allocation across the cluster.

CoreOS Just Got Easier to Try With Panamax

2014-08-21T00:00:00+00:00

This is a guest post by Lucas Carlson, Head of CenturyLink Labs

Here at CenturyLink Labs, we help people learn how to adopt new technologies like Docker and CoreOS into their daily lives. This has given us a unique perspective on the Docker ecosystem because we are trying to stay on top of one of the fastest growing open-source projects in history.

After talking to tens of thousands of developers and ops people, we kept hearing the same thing over and over:

CoreOS and Docker is the most transformative technology we have seen in years, but it is still really really hard to get started.

TL;DR: Try Deploying a CoreOS App on Panamax in Minutes

What is Panamax?

Instead of just blogging and podcasting tutorials and interviews (we do that too), we decided to create an open-source project that made the setup and app-creation process for Docker and CoreOS way way easier.

We call it Panamax–Docker Management for Humans.

Panamax starts with a CoreOS installation and adds a few Panamax containers into the CoreOS system that provide a great UX where you can search the entire Docker Hub for any container you want and pull it into your CoreOS system and stitch it together with other containers.

The Panamax containers don’t get in your way, in fact they just run fleet commands for you but you can always go back and run fleetctl yourself and get under the hood. We just wanted to connect the dots and setup best practices so that to get started you wouldn’t need to spend weeks getting up to speed on new technology.

App Template Repository

The thing we are most excited about with Panamax are the application templates. It is like a Fig specification that can run on top of a clustered CoreOS. Here are just a few we created to seed the community: https://github.com/CenturyLinkLabs/panamax-public-templates and people are building a bunch more right now for a contest we started here: https://github.com/CenturyLinkLabs/panamax-contest-templates.

There are application templates for setting up CoreOS-based apps in one-click for:

Heroku Buildpacks
Minecraft
WordPress
Ghost
Drupal
GitLab
Magento
Shippable
Ngrok

And a bunch more. Most of these templates are built with 12-factor micro-services in mind so they combine multiple containers for you automatically and create the connections for you.

Build Your Own Apps on CoreOS

Panamax isn’t just for deploying pre-baked applications, it is also a web gui to help you create your own apps. In fact, if you create an app and contribute it by making a pull request back to the template repository before Tuesday August 26th, 2014, you can have a chance to win one of 30 new Mac Pros or 30 iPad Airs that CenturyLink is giving away to kick-start the Panamax community.

What Are You Waiting For?

Panamax runs on any cloud that supports CoreOS: Amazon, Rackspace, Google, CenturyLink Cloud and even your laptop. The install process on your Mac to try it couldn’t be easier if you have Homebrew installed already:

brew install http://download.panamax.io/installer/brew/panamax.rb && panamax init

If you are on Linux, check out the installation instructions on GitHub.

If you haven’t used CoreOS yet because you have been intimidated or not sure you were ready to make the commitment yet, now’s the time to give it a try. Panamax will get you up and running with CoreOS is just minutes. Don’t take my word for it, try it yourself!

CoreOS Certification and Training

2014-08-20T00:00:00+00:00

Update: We've updated a few aspects of our training program and the CoreOS Specialist certification is not offered at this time.

Today we are excited to bring you more opportunities to learn about CoreOS - both through the Linux Foundation’s Certification Program that was announced today and through in-person training courses.

CoreOS Certification

In partnership with the Linux Foundation, CoreOS is now offering an online certification class to become a CoreOS specialist. The online exam is a multiple choice exam designed to test your knowledge of CoreOS components, operating principles and troubleshooting. The CoreOS certification can be attempted with or without taking the CoreOS training course, and rewards you with tangible proof of your CoreOS expertise.

Trainings and Seminars

In addition to the certification, we also announced in-person training courses on CoreOS. Individuals can attend five day public training sessions, or work with the CoreOS training team to arrange a custom onsite session tailored to fit company-wide engineering team goals and questions. At the trainings, you’ll get one-on-one time with a CoreOS team member, an overview of installing, configuring, and managing CoreOS and all of its components, as well as the tools to begin deploying and managing applications at scale. The training will also prepare you to pass the CoreOS certification exam.

Quay.io joins CoreOS, Introducing the CoreOS Enterprise Registry

2014-08-13T00:00:00+00:00

Today we announced that Quay.io is now part of the CoreOS family. If you are not familiar with Quay.io, it is the first hosted private docker registry. In addition to continuing to invest in Quay.io, we are launching the CoreOS Enterprise Registry based on Quay.io. Enterprise Registry is a solution for companies that want to run their own docker registry.

Enterprise Registry includes all the bells and whistles, including a rich UI, access control lists, and powerful tools for teams collaborating with containers. It is similar to GitHub Enterprise, in that CoreOS Enterprise Registry provides tools to group together teams within your organization, and assign permissions to them so they can work together securely. This all happens behind your firewall and within your datacenter.

For those of you that want to try it out, you can head over to Quay.io and give the hosted version a shot. For the next 1000 users that sign up, we will be giving away six months of the “Yacht” plan, i.e. 20 free private docker repositories. We think you’ll be very impressed with what the team has built.

Finally, CoreOS could not be more proud to welcome our new team members, Jake Moshenko and Joseph Schorr. Jake and Joseph are a very talented team, whom initially worked together on the Google API platform before starting Quay.io. We are very happy to have them joining CoreOS and kicking off our New York City office.

If you are interested in learning more about Enterprise Registry, Quay.io, or joining the team in NYC, please reach out to us at sales@coreos.com.

Running Kubernetes Example on CoreOS, Part 2

2014-07-30T00:00:00+00:00

Edit: The most up to date Kubernetes + CoreOS guide can be found on the Kubernetes GitHub project.

In the previous post I outlined how to set up a single node Kubernetes cluster manually. This was a great way to get started and try out the basic Kubernetes examples. Now its time to take it up a notch.

In this post I will demonstrate how to get Kubernetes installed on a three node CoreOS cluster running on VMware Fusion. The goal is to setup an environment that closely mimics a cluster of bare-metal machines supporting the Kubernetes networking model.

Before we jump into the tutorial we’ll take a moment to understand what Kubernetes is all about. I tend to think of Kubernetes as a project that builds on the idea that the datacenter is the computer, and Kubernetes provides an API to manage it all.

The deployment workflow provided by Kubernetes allows us to think about the cluster as a whole and less about each individual machine. In the Kubernetes model we don't deploy containers to specific hosts, instead we describe the containers we want running, and Kubernetes figures out how and where to run them. This declarative style of managing infrastructure opens up the door for large scale deployments and self healing infrastructures. Yeah, welcome to the future.

But how does it all work?

Kubernetes introduces the concept of a Pod, which is a group of dependent containers that share networking and a filesystem. Pods are great for applications that benefit from collocating services such as a caching server or log manager. Pods are created within a Kubernetes cluster through specification files that are pushed to the Kubernetes API. Kubernetes then schedules Pod creation on a machine in the cluster.

At this point the Kubelet service running on the machine kicks in and starts the related containers via the Docker API. It’s the Kubelet’s responsibility to keep the scheduled containers up and running until the Pod is either deleted or scheduled onto another machine.

Next Kubernetes provides a form of service discovery based on labels. Each collection of Pods can have an arbitrary number of labels which can be used to locate them. For example, to locate the Redis service running in production you could use the following labels to perform a label query:

service=redis,environment=production

Finally, Kubernetes ships with a TCP proxy that runs on every host. The TCP proxy is responsible for mapping service ports to label queries. The idea is that consumers of a service could contact any machine in a Kubernetes cluster on a service port and the request is load balanced across Pods responsible for that service.

The service proxy works even if the target Pods runs on other hosts. It should also be noted that every Pod in a Kubernetes cluster has its own IP address, and has the ability to communicate with other Pods within the cluster.

If you are interested in the inter-workings of Kubernetes, be sure to checkout the Kubernetes documentation.

Now it's time to build our own Kubernetes cluster.

The Environment

The examples in this post were testing with VMware Fusion 6 on OS X 10.9.4, but should also work with other virtualization products.

The Network

The Kubernetes networking model aims to provide each Pod (container set) with its own IP address and supports cross host communication. To make this work create a custom VMware network, vmnet2, dedicated for containers. The vmnet2 network should have DHCP and NAT disabled to mimic a basic switch.

Next create 3 VMs with the following virtual hardware specs:

1 CPU
512 MB RAM
20 GB HDD
2 Network Interfaces
CD ROM (Used for CoreOS installation and config-drive)

The second network interface for each VM will be connected to the vmnet2 network and will later become a member of the cbr0 bridge used by Docker.

Each VM plays a specific role in the Kubernetes cluster, either a Master or Minion in Kubernetes terms. The Master runs the apiserver, controller manager and optionally the kubelet and proxy servers. Minions on the other hand run only the kubelet and proxy servers. While you can have more than one Minion there can only be a single Master per cluster; a Kubernetes limitation that will removed in the near future.

Next install CoreOS on each of the VMs. For this environment I followed the ISO installation method and performed a local install. At this point you should have a clean set of VMs and are ready to install Kubernetes.

Installing Kubernetes

The recommend way of setting up Kubernetes on CoreOS is via Cloud-Config. The cloud-config files used for this tutorial can be found on GitHub.

There is one cloud-config file for each VM:

master.yml
node1.yml
node2.yml

Using cloud-config files we can automate 100% of the Kubernetes installation and setup process. There are a number of items being setup and configured in each cloud-config file:

Set the hostname
Configure static networking for the primary interface
Setup the cbr0 bridge and assign it a subnet from the 10.244.0.0/16 range
Configure iptables to NAT non-container traffic through the primary interface
Configure Docker to use the cbr0 bridge and disable adding rules to iptables
Download and install the Kubernetes binaries
Install the systemd units for each Kubernetes service
Configure etcd to use a static set of peers

Customize the Cloud-Config Files

The cloud-config files should not be used as is; the following changes must be made:

Change the static host IP address throughout the cloud-config file.
Change the SSH authorized key

Not excited about updating a bunch of cloud-config files? I figured as much, so I wrote kubeconfig to help you out. Instead of editing the cloud-configs manually you can use kubeconfig to generate the cloud-config files specifically for your environment.

wget http://storage.googleapis.com/kubernetes/darwin/kubeconfig -O /usr/local/bin/kubeconfig
chmod +x /usr/local/bin/kubeconfig

Before we can use kubeconfig we need to gather the following bits of information:

Network information to configure static networking on each of the VMs
- DNS server
- Gateway
- 3 IP addresses
SSH Public Key

Gather 3 IP Addresses

First allocate 3 IP addresses to be used by the primary network interfaces of the VMs. For my setup the primary interfaces are connected to the vmnet8 virtual network.

To avoid picking IP addresses within the VMware DHCP range for the vmnet8 network, view the DHCP configuration file with the following command:

cat "/Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf"

On my system I get the following output:

subnet 192.168.12.0 netmask 255.255.255.0 {
  range 192.168.12.128 192.168.12.254;
  option broadcast-address 192.168.12.255;
  option domain-name-servers 192.168.12.2;
  option domain-name localdomain;
  default-lease-time 1800;                # default is 30 minutes
  max-lease-time 7200;                    # default is 2 hours
  option netbios-name-servers 192.168.12.2;
  option routers 192.168.12.2;
}

From this output I can see the DHCP range, DNS (domain-name-servers), and Gateway (routers) I should use in my config.yml file.

In my environment I’ve chosen the following network settings:

dns: 192.168.12.2
gateway: 192.168.12.2
master_ip: 192.168.12.10
node1_ip: 192.168.12.11
node2_ip: 192.168.12.12

Gather the SSH authorized key

Finally set the sshkey key with your own public key. Skipping this step will prevent you from logging into your VMs.

sshkey: ssh-rsa AAAAB3NzaC1yc2..

Your final config.yml file should look something like this:

dns: 192.168.12.2
gateway: 192.168.12.2
master_ip: 192.168.12.10
node1_ip: 192.168.12.11
node2_ip: 192.168.12.12
sshkey: <public ssh key>

To generate the cloud-config files run the following command:

kubeconfig -c config.yml

Create Configuration Drives

While we could use the cloud-config files generated by kubeconfig directly, it’s easier to use config-drives to expose our cloud-configs to the VMs. Again we can use kubeconfig to automate this process for us.

kubeconfig -c config.yml -iso

The result of running the above command is a config-drive for all three VMs:

master.iso
node1.iso
node2.iso

For each VM attach the related config-drive and boot them.

At this point you should be able to login as the core user to each of your VMs. It will take a few minutes for the Kubernetes binaries to download and the related services to start.

Use the systemd journal to monitor progress of the VMs after they boot up:

journalctl -f

You can check the status of each of the Kubernetes components with the following commands

sudo systemctl status apiserver
sudo systemctl status controller-manager
sudo systemctl status kubelet
sudo systemctl status proxy
sudo systemctl status etcd
sudo systemctl status docker

You are now ready to start running the examples hosted on the Kubernetes GitHub repo.

Running Kubernetes Examples

You have two choices on running the examples. You can log on to the Master and run the kubecfg command from there. Or you can download the kubecfg command line tool to your local machine and execute commands remotely.

Conclusion

This post demonstrated how to setup a CoreOS cluster running Kubernetes on virtual hardware that mimics a bare metal infrastructure. Kubernetes consists of many moving parts, but as you can see the installation and setup of everything can be fully automated when using CoreOS with Cloud-Config.

Kubernetes is still in the early stages and things are rapidly evolving. The good news is you’ll have the ability to help shape the direction of the Kubernetes project by kicking the tires and joining the community. CoreOS will be here to support you by providing the best platform for running Linux containers and hosting projects like Kubernetes.

CoreOS Stable Release

2014-07-25T00:00:00+00:00

First off, Happy SysAdmin Day. We think we have a pretty good SysAdmin surprise in store for you today as we are announcing the CoreOS stable release channel. Starting today, you can begin running CoreOS in production. This version is the most tested, secure and reliable version available for users wanting to run CoreOS. This is a huge milestone for us. Since our first alpha release in August 2013:

191 releases have been tagged
Tested on hundreds of thousands of servers on the alpha and beta channels
Supported on 10+ platforms, ranging from bare metal to being primary images on Rackspace and Google

It is a big day for us here at CoreOS, as we have been working hard to deliver the stable release. Of course, we couldn’t do this without the community so thank you for all of your support and contributions to the project.

CoreOS 367.1.0, our first version on the stable channel, includes the following:

Linux 3.15.2
Docker 1.0.1
Support on all major cloud providers, including Rackspace Cloud, Amazon EC2 (including HVM), and Google Compute Engine
Commercial support via CoreOS Managed Linux

This is a great opportunity to read about our Update Philosophy if you haven't already done so.

Please note: The stable release is not including etcd and fleet as stable, this release is only targeted at the base OS and Docker 1.0. etcd/fleet stable support will be in subsequent releases.

For those of you who want to start running CoreOS in production be sure to review our quick Switching Release Channels guide. As you're booting new machines, be sure to base them off your desired channel from the beginning.

Finally, thanks to the community for your support. We can’t wait to hear your feedback. For those looking for additional support of running CoreOS in production, be sure to check out our Managed Linux offerings, as we have a full support team in place ready to answer any questions you may have.

Happy SysAdmin Day, and thank you for making the web awesome.

Running Kubernetes Example on CoreOS, Part 1

2014-07-10T00:00:00+00:00

Edit: The most up to date Kubernetes + CoreOS guide can be found on the Kubernetes GitHub project.

This post is not kept up to date with the latest developments! The latest documentation can be found at this GitHub repository: https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-started-guides/coreos/units

Kubernetes is the container cluster manager from Google that offers a unique workflow for managing containers across multiple machines. Kubernetes introduces the concept of a pod, which represents a group of containers that should be deployed as a single logical service. The pod concept tends to fit well with the popular pattern of running a single service per container. Kubernetes makes it easy to run multiple pods on a single machine or across an entire cluster for better resource utilization and high availability. Kubernetes also actively monitors the health of pods to ensure they are always running within the cluster.

Kubernetes is able to achieve this advanced level of cluster wide orchestration by leveraging the etcd distributed key-value store; a project started and maintained by CoreOS. etcd takes care of storing and replicating data used by Kubernetes across the entire cluster, and thanks to the Raft consensus algorithm etcd can recover from hardware failure and network partitions. The combination of Kubernetes and etcd is even more powerful when deployed on CoreOS, an operating system built for this kinda thing.

Now lets see Kubernetes in action.

Feel free to follow along at home as we step through getting Kubernetes installed on CoreOS and bringing up a single pod running a Redis database. First step is to get CoreOS up and running; a single node cluster will work for this tutorial. Take a look at the CoreOS installation docs and choose your favorite platform. Next we need to download the Kubernetes components onto our CoreOS instance.

For convenience we've precompiled all the Kubernetes components including:

apiserver
controller-manager
kubecfg
kubelet
proxy

sudo mkdir -p /opt/kubernetes/bin 
sudo chown -R core: /opt/kubernetes
cd /opt/kubernetes
wget https://github.com/kelseyhightower/kubernetes-coreos/releases/download/v0.0.1/kubernetes-coreos.tar.gz
tar -C bin/ -xvf kubernetes-coreos.tar.gz

At this point you should have all the Kubernetes components installed under the /opt/kubernetes/bin directory. Kubernetes also requires Docker and etcd, both ship with CoreOS out of the box so nothing to do here. While CoreOS recommends that you run third party applications via Docker containers, CoreOS does support running Kubernetes directly on the OS. The best way to do that is by creating systemd unit files and starting the Kubernetes components with systemctl.

git clone https://github.com/kelseyhightower/kubernetes-coreos.git
sudo cp kubernetes-coreos/units/* /etc/systemd/system/

Before starting everything, make sure etcd is started:

sudo systemctl start etcd

With the systemd units in place we are ready to start the Kubernetes services.

sudo systemctl start apiserver
sudo systemctl start controller-manager
sudo systemctl start kubelet
sudo systemctl start proxy

Congratulations! You now have Kubernetes up and running on CoreOS. Take a moment and tweet that!

Creating a Kubernetes pod

Pods are how Kubernetes groups containers and define a logical deployment group. Lets take a look at a simple pod configuration for a Redis database.

{
  "id": "redis",
  "desiredState": {
    "manifest": {
      "version": "v1beta1",
      "id": "redis",
      "containers": [{
        "name": "redis",
        "image": "dockerfile/redis",
        "ports": [{
          "containerPort": 6379,
          "hostPort": 6379 
        }]
      }]
    }
  },
  "labels": {
    "name": "redis"
  }
}

You’ll notice that a pod configuration describes a Docker container and port mapping. Pods also define a label which is used for service discovery. We’ll cover service discovery in more detail in the next post, but for now know that it’s possible for other tools to discover the pods you spin up using Kubernetes.

Finally we can create the Redis database pod with the kubecfg command line tool and the Redis pod configuration file.

/opt/kubernetes/bin/kubecfg -h http://127.0.0.1:8080 -c kubernetes-coreos/pods/redis.json create /pods

(this command might take a minute to complete depending on how fast the docker pull is)

At this point the Redis database pod is up and running. We can validate this with a Redis client and connecting to our pod through the docker0 interface.

docker run -t -i dockerfile/redis /usr/local/bin/redis-cli -h 172.17.42.1

You can also use the kubecfg command to list running pods.

/opt/kubernetes/bin/kubecfg -h http://127.0.0.1:8080 list /pods

Once you are done with a pod you can simply delete it.

/opt/kubernetes/bin/kubecfg -h http://127.0.0.1:8080 delete /pods/redis

Conclusion

Hopefully this blog post has demonstrated that Kubernetes can be installed outside of GCE onto just about any Linux platform. CoreOS makes it simple to setup Kubernetes since it ships with Docker, etcd and systemd out of the box. All you have to do is install a few binaries and systemd units then you are ready to go.

In next post we will take a look at the true power of Kubernetes, which is managing Linux containers across multiple machines, service discovery, and full life cycle management of your Linux containers.

* All code examples, systemd units, and precompiled binaries can be found at the following github repository: https://github.com/kelseyhightower/kubernetes-coreos

The CoreOS Epoch

2014-06-30T00:00:00+00:00

CoreOS has a rapidly incrementing version, 197.0.0, 247.0.0, and so on. It is a little known fact, but the version represents the number of days since the CoreOS epoch on July 1, 2013. And we hope to make 365.0.0 a particularly special release. Over the next week this release will go through our alpha and beta channels and, if all goes well, it will be the first version to be promoted to stable. So, watch along with us in the release channel to see if this is the one!

We’ve also used this opportunity to announce our Series A funding and two new products:

CoreOS Managed Linux is for companies looking for support as they implement CoreOS in their infrastructure.
CoreUpdate is a new product, and provides a dashboard for full control of rolling updates.

If you are waiting for etcd and fleet support, keep an eye out as we will be announcing more soon on this front with CoreOS Cluster.

Given all the celebration, we felt it was an appropriate time to send a big thanks to the community, which has grown with us throughout the past year. Some of our highlights over the past year include:

Over 150 git contributors to our GitHub projects
5,000+ GitHub stars collectively on CoreOS projects
CoreOS has contributed features and fixes to open source projects including: Docker, Linux Kernel, networkd, systemd and more
Official CoreOS image added to Google Compute Engine, Rackspace, Amazon
Joining the Docker Governance Board as a Contributing Member
Today’s most respected technology companies and many Fortune 500 companies are using and testing CoreOS in their environments

If you are in the mood to celebrate the CoreOS epoch this Tuesday and happen to be on either coast, join us for a toast. In San Francisco, we are having our meetup on Tuesday, July 1st, 6-9 pm at Geekdom San Francisco. And if you happen to be in New York, Alex Polvi, co-founder and CEO will host a drink up celebration at Sweet and Vicious, 4-7:30 pm, for the USA vs Belgium soccer game. First drink is on CoreOS.

Thanks for your help over the past year and set your timers, the countdown to stable has begun. Once available the stable channel will be available to all users just like the alpha and beta channels. Users are encouraged to have some machines in each channel to test out changes coming to their infrastructure and to help the community find and fix bugs.

CoreOS Officially on Rackspace OnMetal Cloud Servers

2014-06-19T00:00:00+00:00

Earlier this morning at Structure, Rackspace announced OnMetal, their API-driven, single-tenant infrastructure-as-a-service offering. We are excited to announce that CoreOS will be available at launch!

Rackspace OnMetal Cloud Servers is the first of its kind. Using OnMetal you create machine instances using OpenStack APIs but, instead of being provisioned a virtual machine, you are provisioned physical hardware. All of this happens with the speed and ease you’ve come to expect for cloud deployments - on demand.

If you are tracking upstream OpenStack you may have noticed commits related to CoreOS. The OpenStack Ironic project and CoreOS are key components of the technology making OnMetal possible. Rackspace’s continued development of OpenStack demonstrates their commitment to the open source community and we are grateful to be involved in such a fundamental shift in on-demand computing.

Once OnMetal is live, you'll be able to provision instances on Rackspace's OnMetal cloud and run CoreOS. This combination gives you the ease of spinning up machines with the click of a mouse, the security of running CoreOS with automatic updates, and the raw horsepower of running on bare metal.

The CoreOS and Rackspace teams have worked very closely to ensure that CoreOS and OnMetal are a solid combination and we are very excited to make this available to you.

If you happen to be at Structure today and have any questions about CoreOS on Rackspace OnMetal, stop by the Rackspace lounge area and ask for Redbeard or tweet at us @coreoslinux. We are happy to answer any of your questions.

The CoreOS Update Philosophy

2014-06-18T00:00:00+00:00

Just a few hours after the Docker team announced Docker 1.0, CoreOS shipped the update to all users on the alpha channel. If you are using CoreOS Alpha with automatic updates enabled, you were moved to Docker 1.0 seamlessly. The recent Docker exploit is just a small example of a long list of vulnerabilities that have plagued server software since it has existed, and why the CoreOS model is extremely important.

Running the latest software traditionally meant taking on risks around stability, security, and incompatibility. Times are changing, and our server deployments need to adapt. Software development is now at an arms race against software exploitation. Yet for many organizations the process of patching is still too difficult and time consuming, and is often neglected as a result. CoreOS and our Update Philosophy aims to remove these pain points and enable our users to safely, and easily, always run the latest, most secure, and stable versions of the software we support, without breaking your deployment.

CoreOS Videos From Our Inaugural Meetup

2014-06-17T00:00:00+00:00

At our inaugural meetup here in San Francisco, our host Geekdom SF, was kind enough to film and edit our talks for your viewing pleasure. We know many of you have been asking about the talks so without further ado here are the talks in all their glory.

CoreOS: Anatomy of a CoreOS Update

Presented by Brian Redbeard Harrington

CoreOS: Orchestrating the Fleet

Presented by Brian Waldon

Running CoreOS Outside of the Cloud with iPXE

Presented by Kelsey Hightower

If you would like to hear more about CoreOS or would like to ask us questions in person, you are in luck. Tomorrow, we will be at the AWS Pop-up Loft here in San Francisco from 6-7:30 pm giving an intro to CoreOS, discussing how startups use CoreOS for infrastructure that scales and going over how to set up CoreOS on EC2. RSVP on the AWS eventbrite to attend.

We are also part of a special announcement this Thursday with Rackspace, and will be attending their party Thursday evening. If you prefer to hang out with us there, then RSVP to their “Hands On Heavy Metal” party.

Enjoy the videos and we hope to see you at one of the events this week.

Docker 1.0 released to Alpha

2014-06-16T00:00:00+00:00

Try Out Docker 1.0 In The CoreOS Alpha Channel! If you're new to Docker, check out our Getting Started with docker guide.

Last Monday Docker 1.0 was announced at DockerCon, and shortly after Docker 1.0 was made available in the CoreOS alpha channel.

Want to try out Docker 1.0? CoreOS has it in the alpha channel. https://t.co/liT6z73pPu
— CoreOS Linux (@coreoslinux) June 9, 2014

Docker 1.0 is the the stable, production-ready version of Docker. A fundamental building block of CoreOS is the use of Linux containers, where your applications and code run. A natural choice for Linux containers was Docker due to their expansive ecosystem. The Docker engine is installed on each CoreOS machine.

If you want to try out Docker, you can package each component of your application inside of a container (web server, caching, database) and then schedule them to a cluster with fleet. Distributed configuration and locking become trivial additions by connecting all of your components together with etcd.

If you are interested in trying out Docker 1.0, get to our alpha channel and try it out first hand. Or if you have questions, join us in #coreos on IRC or email the dev list.

Official CoreOS Meetup in San Francisco June 3rd, 2014

2014-05-28T00:00:00+00:00

Join us for our first CoreOS meetup Tuesday, June 3rd at the San Francisco Geekdom meeting space. Come with your questions queued up and ready to learn as we'll be hosting two separate presentations geared at both beginners and advanced users.

Anatomy of a CoreOS update, a look behind the curtain of your update process

Brian Harrington

CoreOS is a self-updating operating system, but someone produces these updates. We will take some time to discuss the Omaha protocol, what an update does when it's downloaded and applied, the benefit of keeping a locksmith on hand, and why you can trust a third party with your data.

Orchestrating the fleet, how a systemd unit achieves resilience

Brian Waldon

Fleet is a distributed init toolkit for building resilient systemd units. We'll take the time to show you both the ways these processes are resillient (and not), scheduling units across your cluster, and the road ahead for managing your fleet of containers.

After the main presentations members of the team will be available for birds of a feather sessions to discuss etcd, fleet, and the operating system. This is a great chance to learn what others are doing and get realtime feedback.

Make sure you RSVP here as space is limited.

Official CoreOS Images on Google Compute Engine

2014-05-23T00:00:00+00:00

Today we're excited to announce that official CoreOS images are available on Google Compute Engine. This means it's now even easier to spin up a CoreOS cluster on GCE using the API or from the command line. Adding an instance is as simple as:

gcutil addinstance --image=projects/coreos-cloud/global/images/coreos-alpha-317-0-0-v20140515 core1

In the next few days, CoreOS will become available as a default image type in the GCE control panel, making running your first CoreOS cluster on GCE an easy, browser-based experience. They will also become globally available to users via gcutil listimages.

CoreOS is an ideal host for distributed systems and Google Compute Engine is a perfect base for CoreOS clusters. Google runs industry-leading distributed systems and their expertise shows in the speed and stability of Google Compute Engine. One feature that we were particularly delighted to see was a shared project metadata store that looks very much like our own etcd.

GCE platform tools including load balancers and replica pools are supported by the CoreOS GCE images and enable you to efficiently and smoothly scale your cluster. After grabbing user data and SSH keys from GCE, newly-booted CoreOS machines come online and immediately join the cluster, and fleet will automatically begin placing qualified jobs on these new machines.

For more details, hop over to the Google Cloud Platform blog.

etcd 0.4.0 with Standby Mode

2014-05-20T00:00:00+00:00

The etcd team has been focused on making it easier to scale and manage larger clusters, and is happy to announce a release with features to help: etcd v0.4.0. This release is an important step in our road to 1.0. (If you are new to etcd, our getting started guide can give you a quick overview of the project).

This release is will be available in the alpha channel of CoreOS in the next few days, and will roll out to the beta channel after it has proven solid.

What's New in 0.4.0

Cluster Management API

This release introduces a documented API for removing machines from a cluster. This gives administrators the ability to remove downed machines from the cluster so that new machines can join.

This new endpoint also includes options for controlling two important new cluster-wide features: standby mode, and automatic node promotion/demotion.

Standby Mode

The new "standby mode" adds extra resiliency to etcd. All machines in an etcd cluster are now in one of either two modes: peer mode or standby mode. Peers participate in the Raft consensus algorithm; in earlier releases of etcd, every node in a cluster was a peer. Standbys, on the other hand, do not participate in consensus, and instead redirect client requests to participating peers. This setup enables users to have a small number of machines participating in consensus directly while having a much larger number of standbys that are keeping up to date on the cluster members.

You can find more details about standby mode here.

Node Promotion/Demotion

To complement standby mode, etcd can automatically reconfigure the cluster by promoting and demoting nodes between standby and peer modes. As part of the cluster configuration, every cluster now has a defined maximum number of participating peers (activeSize). Any etcd process that attempts to join a cluster greater than or equal to the maximum number of peers will start out as a standby. If the number of peers in the cluster drops below the configured activeSize, then nodes in standby will automatically be promoted to peers until the activeSize is reached.

You can learn more about this feature in the API documentation.

HEAD Requests

Clients can now use HEAD to check for the existence of a key without a body. This is useful for avoiding unnecessary network costs when the body of the request isn’t required.

Updated Peer Discovery

When an instance starts up, etcd now uses the following sources, in order, to locate other peers in the cluster:

log data in the -data-dir
-discovery endpoints
a static -peers list

This improves on v0.3.0 and promotes the priority of log data before finding peers from other sources. In practice this means it's easier for an etcd instance to reconnect to a cluster of which it was previously a member.

Name IP Migration

If a machine's address is changed, etcd will now accept the new address. This is useful in the case of new DHCP leases or virtual machine migrations.

Deprecation of /mod Modules

Modules have been a great experimental testing ground for higher level features built on top etcd. Until now, they have seen some use but haven’t had consistent stability. We are focusing on improving the etcd core API and will turn back to these experimental modules in a later future release. It was not an easy decision, but we are sure you will be happy with what results this decision can allow us to produce.

Minor Fixes and Improvements

Along with the features and fixes outlined above we have made a number of smaller improvements:

Set logs NOCOW flag when BTRFS is detected to avoid fsync overhead
Fix all known data races, and pass Go race detector
Fixed timeouts when using HTTPS
Improved snapshot stability

Mailing List

Please join the etcd development mailing list to discuss etcd with us!

Update: 0.4.1 has been released.

Zero Downtime Frontend Deploys with Vulcand on CoreOS

2014-05-19T00:00:00+00:00

Update: Vulcand has been updated since this post was written. Jump to vulcanproxy.com for more info. You can follow the concepts of this post, but the commands are no longer up to date.

Running a distributed system across many machines requires a sophisticated load balancing mechanism that can reconfigure itself quickly and reliably.

The team over at Mailgun has built vulcand, an etcd-backed load balancer, to serve traffic to different parts of their systems. vulcand has many awesome features such as an HTTP API, a command line utility and the ability for complex routing rules. We're only going look at a few simple examples in this post, but you can check out the readme for the complete details.

Today we're going to deploy Vulcan on a CoreOS cluster and use it to facilitate two strategies for zero-downtime deployments. These examples were intentionally kept simple for easy comprehension and it's up to you to decide what strategies you want to use for high availablity, port management, etc.

The Set Up

This post is going to cover two common front-end deployment strategies: a rolling-upgrade and a rapid switch to a new software version. We're going to assume you're familar with CoreOS, docker and fleet, and also have a 3 node CoreOS cluster to run the examples on. This post was tested on CoreOS 317.0.0 with etcd 0.2 and fleet 0.3.2.

Our systemd units are going to use containers that are located on the public docker index as coreos/example and have been tagged as 1.0.0 and 2.0.0. Mailgun has set up a trusted build for vulcand as mailgun/vulcand.

Each unit contains a web server that serves a very simple webpage. To easily tell the difference between versions, 1.0.0 has a red background and 2.0.0 has a blue background.

Let's get started.

Clone the Example Units Repository

The unit-examples repository contains all of the example units used in our blog posts. Clone it to save yourself from having to copy/paste everything:

git clone https://github.com/coreos/unit-examples.git
cd unit-examples
cd blog-vulcan-example

If you're planning on submitting units to the cluster remotely, your local copy of fleetctl must match the version of fleet running on the CoreOS machines. You can find the versions with fleetctl -version and fleet -version. Browse the tagged releases on GitHub for both Linux and Mac versions of fleetctl.

Start Vulcan

First, start up an instance of vulcand and configure a DNS record to point to it. If you're using Vagrant, you may need to forward ports on your laptop to Vagrant.

You'll need to modify the unit files in this example with the domain/subdomain you're using in order for vulcan's routing to work properly. The easiest way to do this is with sed after you've cloned the units repository:

find ./ -type f -exec sed -i -e 's/example.com/vulcan.test.company.com/g' {} \;

vulcan.service

We're going to use the "trusted build" of vulcand on the public index. The trusted build is a container that is built directly from the vulcand repository every time code is commited to master. This ensures that the code you're running comes directly from Mailgun and is always up to date.

You can also clone the vulcand repository on your CoreOS machine and build the Dockerfile manually if you'd prefer. Here's the unit we're going to use:

[Unit]
Description=Vulcan
After=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill vulcan1
ExecStartPre=-/usr/bin/docker rm vulcan1
ExecStartPre=/usr/bin/docker pull mailgun/vulcand:v0.7.0
ExecStart=/usr/bin/docker run --rm --name vulcan1 -p 80:80 -p 8182:8182 mailgun/vulcand:v0.7.0 /go/bin/vulcand -apiInterface=0.0.0.0 -interface=0.0.0.0 -etcd=http://172.17.42.1:4001 -port=80 -apiPort=8182
ExecStop=/usr/bin/docker kill vulcan1

The -etcd flag tells vulcand that our etcd cluster can be reached over the docker0 bridge since we're running vulcand in a container. On Vagrant machines, this IP can be different and you may need to edit the unit. Let's start it:

$ fleetctl start vulcan.service
Job vulcan.service scheduled to f8ea00d4.../10.0.2.15

To test it out, load up the location in a browser. You should see error: "Bad Gateway" since we haven't set up anything to receive traffic. If you don't see anything, check to see if the units have started successfully:

$ fleetctl list-units
UNIT    STATE   LOAD  ACTIVE  SUB DESC  MACHINE
vulcan.service  launched  loaded  active  running Vulcan  7f0e81ab.../10.0.2.15

If it's activating and start-pre it means that the docker pull from the ExecStartPre is still running. Check the journal of a unit to confirm this:

$ fleetctl journal vulcan.service
May 12 20:59:38 core-01 systemd[1]: Started Vulcan.
May 12 20:59:39 core-01 bash[3268]: 2014/05/12 20:59:39 Error: No such container: vulcan1
May 12 20:59:39 core-01 bash[3268]: Unable to find image 'coreos/vulcan' locally
May 12 20:59:39 core-01 bash[3268]: Pulling repository coreos/vulcan

If your terminal supports split panes, it's useful to run watch -n 10 fleetctl list-units in a new pane so you can keep an eye on what's happeneing. Feel free to read on while the container downloads.

How Vulcand Works

vulcand has two main concepts, locations and upstreams, that are connected together to serve traffic. A location is a combination of a hostname and a path that can be matched with a regular expression. The location is matched up with an upstream, which is a group of endpoints that are qualified to serve a subset of traffic. All of the examples for manuipulating vulcand in this post will be shown using etcdctl, but an HTTP API and command line tool are also provided.

Set Up the Location

Before we can start routing traffic, we need to set up the location. In the provided units, this is done in the registration sidekick in order to create the location if no units are already running. Here's a breakdown of the commands contained in our registration unit. You don't have to run any of these, they are just for illustration:

This will create the hostname example.com and the path, home, to be matched by the regular expression /.*, which should match all paths:

etcdctl set "/vulcand/hosts/example.com/locations/home/path" '/.*'

Next, we need to register our container as a member of the upstream example. We're using the unit name as the unique identifier:

etcdctl set "/vulcand/upstreams/example/endpoints/mixed-register-v1.0.0-A.service" http://10.10.10.10:8086

The last step is to tell our location to use our upstream:

etcdctl set "/vulcand/hosts/example.com/locations/home/upstream" example

When one of our containers is stopped we need to deregister it from the upstream:

etcdctl rm "/vulcand/upstreams/example.com/endpoints/mixed-register-v1.0.0-A.service"

Let's add all of this together into a working example.

Scenario 1: Rolling Frontend Update

Our first deployment scenario is very common: a rolling upgrade. This strategy is common if the changes that you're making are hidden through feature flags or aren't going to harm a user's experience if they get routed to mixed versions during a session.

Start Version 1.0.0

Before we start our containers, let's take a look and see what they're doing. We're running our simple web server in example-v1.0.0-*.service and a sidekick registration service in mixed-register-v1.0.0-*.service:

example-v1.0.0-A.service

In the ExecStartPre we're doing a docker pull in order to prevent the unit from reporting active until the container download is complete. If we didn't do this, the registration unit would start before the download is complete. TimeoutStartSec=0 disables systemd's built-in time out when starting a unit, since our docker container is quite large and takes a few minutes to download.

Our unit conflicts with the other 1.0.0 example units in order for them to be spread across machines in the cluster.

[Unit]
Description=Example 1.0.0
After=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill example-1A
ExecStartPre=-/usr/bin/docker rm example-1A
ExecStartPre=/usr/bin/docker pull coreos/example:1.0.0
ExecStart=/usr/bin/docker run --rm --name example-1A -p 8086:80 coreos/example:1.0.0
ExecStop=/usr/bin/docker kill example-1A

[X-Fleet]
X-Conflicts=example-v1.0.0-*.service

mixed-register-v1.0.0-A.service

The registration unit executes the etcdctl commands covered earlier. Including EnvironmentFile=/etc/environment allows us to reference $COREOS_PUBLIC_IPV4 and $COREOS_PRIVATE_IPV4 in our unit. This file is populated on platforms in which CoreOS can determine the network environment (cloud providers, vagrant, etc). RemainAfterExit=yes (docs) will allow this unit to be active even after ExecStart commands run. If it didn't our ExecStop commands would run immediately afterwards.

[Unit]
Description=Register for Example
BindsTo=example-v1.0.0-A.service
After=example-v1.0.0-A.service

[Service]
EnvironmentFile=/etc/environment
RemainAfterExit=yes
ExecStart=/bin/sh -c "/bin/etcdctl set \"/vulcand/upstreams/example/endpoints/mixed-register-v1.0.0-A.service\" http://$COREOS_PUBLIC_IPV4:8085; \
  /bin/etcdctl set \"/vulcand/hosts/example.com/locations/home/path\" '/.*'; \
  /bin/etcdctl set /vulcand/hosts/example.com/locations/home/upstream example"
ExecStop=/bin/sh -c "/bin/etcdctl rm /vulcand/upstreams/example/endpoints/mixed-register-v1.0.0-A.service"

[X-Fleet]
X-ConditionMachineOf=example-v1.0.0-A.service

Start the two 1.0.0 units and their sidekicks:

$ fleetctl start example-v1.0.0-{A,B}.service mixed-register-v1.0.0-{A,B}.service
Job example-v1.0.0-A.service launched on 7f0e81ab.../10.0.2.15
Job mixed-register-v1.0.0-A.service launched on 7f0e81ab.../10.0.2.15
Job example-v1.0.0-B.service launched on 62b05884.../10.0.2.15
Job mixed-register-v1.0.0-B.service launched on 7f0e81ab.../10.0.2.15

The registration unit will create the location and the upstream and vulcan will automatically start sending traffic to it.

When you load the vulcand container in a browser, you should see a red background and the text 1.0.0. As before, you might need to wait for the container to download.

Start Version 2.0.0

Suppose that we've just implemented a great new feature (a blue background!) and we're ready to deploy it. First, build and push the updated image to the docker index. Since we're using the public index this has already been done for you, but here are the commands used:

docker build -t coreos/example:2.0.0 .
docker push coreos/example:2.0.0

Next, we need to launch units that refer to the 2.0.0 tag:

example-v2.0.0-A.service

All references to 1.0.0 have been changed to 2.0.0 and the port has been incremented to allow 1.0.0 and 2.0.0 units to run on the same machine.

[Unit]
Description=Example 2.0.0
After=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill example-2A
ExecStartPre=-/usr/bin/docker rm example-2A
ExecStartPre=/usr/bin/docker pull coreos/example:2.0.0
ExecStart=/usr/bin/docker run --rm --name example-2A -p 8086:80 coreos/example:2.0.0
ExecStop=/usr/bin/docker kill example-2A

[X-Fleet]
X-Conflicts=example-v2.0.0-*.service

Now, start the 2.0.0 units with fleet:

$ fleetctl start example-v2.0.0-{A,B}.service mixed-register-v2.0.0-{A,B}.service
Job mixed.example.1.service scheduled to 99375570.../54.81.79.219
Job mixed.example.2.service scheduled to 5dae80fd.../184.73.78.150
Job mixed.register.1.service scheduled to 99375570.../54.81.79.219
Job mixed.register.2.service scheduled to 5dae80fd.../184.73.78.150

In your browser you should now see the background alternate between red and blue since we're now running both versions of our container concurrently.

Stopping Version 1.0.0

To complete our deployment, destroy the old units with fleet:

$ fleetctl destroy example-v1.0.0-{A,B}.service mixed-register-v1.0.0-{A,B}.service
Destroyed Job example-v1.0.0-A.service
Destroyed Job example-v1.0.0-B.service
Destroyed Job mixed-register-v1.0.0-A.service
Destroyed Job mixed-register-v1.0.0-B.service

Load up the browser a few more times and you should only see the blue background. Our deployment from 1.0.0 → 2.0.0 is now complete. To clean up before our next example, destroy the remaining units and remove all of the state from etcd:

$ fleetctl destroy example* mixed-register*
Destroyed Job example-v1.0.0-A.service
Destroyed Job example-v1.0.0-B.service
Destroyed Job example-v2.0.0-A.service
Destroyed Job example-v2.0.0-B.service
Destroyed Job mixed-register-v1.0.0-A.service
Destroyed Job mixed-register-v1.0.0-B.service
Destroyed Job mixed-register-v2.0.0-A.service
Destroyed Job mixed-register-v2.0.0-B.service

$ etcdctl rm /vulcand --recursive
Cannot print key [/vulcand: Is a directory]

Scenario 2: Rapid Switch to New Version

Another common deployment scenario is the need to rapidly switch from one version of your frontend to another. For this scenario we're going to use the same primary units, but different registration sidekicks for 2.0.0. Instead of registering to the same endpoint, we're going to create one with a different name, example2. Since we need to switch upstreams after our new containers have been deployed, that line has been omitted from our sidekicks.

switch-register-v1.0.0-A.service

[Unit]
Description=Register for Example
BindsTo=example-v1.0.0-A.service
After=example-v1.0.0-A.service

[Service]
EnvironmentFile=/etc/environment
RemainAfterExit=yes
ExecStart=/bin/sh -c "/bin/etcdctl set \"/vulcand/upstreams/example/endpoints/switch-register-v1.0.0-A.service\" http://$COREOS_PUBLIC_IPV4:8085; \
  /bin/etcdctl set \"/vulcand/hosts/example.com/locations/home/path\" '/.*'; \
  /bin/etcdctl set /vulcand/hosts/example.com/locations/home/upstream example"
ExecStop=/bin/sh -c "/bin/etcdctl rm /vulcand/upstreams/example/endpoints/switch-register-v1.0.0-A.service"

[X-Fleet]
X-ConditionMachineOf=example-v1.0.0-A.service

Start Version 1.0.0

Start the units with fleet:

$ fleetctl start example-v1.0.0-{A,B}.service switch-register-v1.0.0-{A,B}.service
Job mixed.example.1.service scheduled to 99375570.../54.81.79.219
Job mixed.example.2.service scheduled to 5dae80fd.../184.73.78.150
Job mixed.register.1.service scheduled to 99375570.../54.81.79.219
Job mixed.register.2.service scheduled to 5dae80fd.../184.73.78.150

You should see a red background in your browser as the requests are sent to each backend:

Start Version 2.0.0

Now start the 2.0.0 units:

$ fleetctl start example-v2.0.0-{A,B}.service switch-register-v2.0.0-{A,B}.service
Job mixed.example.1.service scheduled to 99375570.../54.81.79.219
Job mixed.example.2.service scheduled to 5dae80fd.../184.73.78.150
Job mixed.register.1.service scheduled to 99375570.../54.81.79.219
Job mixed.register.2.service scheduled to 5dae80fd.../184.73.78.150

Since we haven't switched the upstream yet, you should still see the red background.

Switch the Upstream

We need to tell vulcand to reconfigure to use our set of 2.0.0 servers. We do this by switching the upstream by changing the key in etcd. Remember to use the correct hostname:

$ etcdctl set /vulcand/hosts/vulcan.test.company.com/locations/home/upstream example2
example2

You should now see a blue background as all traffic is shifted to our new containers. The switch is now complete.

Clean up the 1.0.0 containers:

$ fleetctl destroy example-v1.0.0-{A,B}.service switch-register-v1.0.0-{A,B}.service
Destroyed Job example-v1.0.0-A.service
Destroyed Job example-v1.0.0-B.service
Destroyed Job mixed-register-v1.0.0-A.service
Destroyed Job mixed-register-v1.0.0-B.service

Next Steps

If you're looking to put a system like this into production, it should be possible to achieve high availability with multiple vulcand containers behind a cloud load balancer, round-robin DNS or another common practice. More complex port management is another issue to tackle if you're running many containers on your cluster.

More Information

To read the complete vulcand documentation, head over to the GitHub page. Be aware that the current status is "Moving fast, breaking things. Will be usable soon, though.". More information about advanced unit files and example fleet deployments can be found in the CoreOS docs.

If you have questions, concerns or improvements to this this vulcand workflow, let us know on the CoreOS user mailing list.

CoreOS Beta Release

2014-05-09T00:00:00+00:00

After 150 releases and 9 months in alpha, we are excited to announce the first CoreOS beta. Starting today, we will be releasing both alpha and beta versions of CoreOS. The beta means you can expect CoreOS itself to not change in any significant way, and it should be considered close to production-ready.

The beta includes a new feature, Locksmith, giving you control over the CoreOS update process, as well as a variety of bug fixes and feature enhancements.

Locksmith

Locksmith gives you complete control over when CoreOS is rebooted after an update is applied. The most advanced feature is the ability to acquire a distributed lock using etcd, before a reboot is applied. This means that you can guarantee, thanks to the consensus algorithms in etcd, that only one machine at a time will be upgraded via reboot. This is critical for rolling an update to all your machines without taking down the cluster. If you use fleet for deploying services, this also means that your processes will be seamlessly migrated around the cluster as the update is performed. Since CoreOS reboots extremely fast (1-2s), you can roll a large cluster in a matter of minutes.

Configuring locksmith

The easiest way to configure Locksmith is via Cloud-Config. The reboot-strategy option determines what action Locksmith takes after an update has been applied to the system. The following strategies are supported:

reboot: Reboot immediately after an update is applied.
etcd-lock: Reboot after first taking a distributed lock in etcd. This guarantees that only one host will reboot concurrently and that the cluster will remain available during the update.
best-effort - If etcd is running, "etcd-lock", otherwise simply "reboot". (default)
off - Do not reboot after updates are applied.

Example cloud-config.yml:

#cloud-config
coreos:
  update:
    reboot-strategy: etcd-lock

Full Locksmith Documentation

View the full Locksmith docs for more information.

How Updates Work on CoreOS

CoreOS uses update channels, similar to client software applications like the Chrome browser. Updates are first shipped to our alpha channel, then if testing is successful, the update is promoted bit-for-bit to the beta channel. Essentially, every alpha is a release candidate for a beta, and every beta is a release candidate for a stable release.

As an example of this in action, the alpha channel will soon be updated to Docker 0.11. Docker 0.10 is still on the beta channel, but will be promoted to 0.11 once our alpha testing has been completed. In general, we expect beta to lag behind alpha by approximately one week. If you want the latest features, use the alpha. If you want stability and feature completeness, use the beta.

Switching Channels

If you're getting closer to running CoreOS in production (and some of you already are), you're probably wondering how to switch your alpha channel machines over to the beta channel. It's really easy and we've got a quick Switching Release Channels guide for you. As you're booting new machines, be sure to base them off your desired channel from the beginning.

Roadmap to Stable

Today's beta release gets us much closer to a stable, production-ready version of CoreOS. The primary work before the stable release is focused on stability of fleet and etcd. Do not expect any major new features - more likely the removal of unused ones.

New coreos-user Mailing List

Primary discussion of CoreOS will move to the new coreos-user list, and we invite you to join us there for discussion and questions around using CoreOS. CoreOS development discussion will remain on the existing coreos-dev list.

Finally, thank you to the incredible community that has grown with CoreOS. We are committed to making CoreOS the best platform of its kind, and could not do it without the help from all of our alpha testers.

Clustering CoreOS with Vagrant

2014-04-24T00:00:00+00:00

When you're first exposed to a distributed platform like CoreOS, running a development environment that is complex enough to match production seems like an impossible task. This article is going to show you how convienient it is run a small CoreOS cluster on your local machine with Vagrant that will mirror the way your production machines are set up.

CoreOS's implementation of cloud-init, coreos-cloudinit, makes it super easy to bootstrap a cluster of CoreOS machines by providing a mechanism for initial machine configuration. The same file can be used to bootstrap CoreOS on Vagrant, Amazon EC2, Google Compute Engine (GCE), and more, making it easy to share the same configuration locally and in production.

This article will focus on Vagrant for a fast, simple deploy on your local machine.

Getting Set Up

Before we get started make sure that you have recent versions of Virtualbox, Vagrant and git installed on your machine.

Now that your local machine is set up for Vagrant, let's grab the CoreOS Vagrant configuration and customize the cloud-config "user-data" file. On OpenStack, EC2 or GCE this would be the same user-data you can provide when starting a virtual machine.

$ git clone https://github.com/coreos/coreos-vagrant
$ cd coreos-vagrant
$ cp user-data.sample user-data

Configure User-Data

Now, generate a new etcd discovery URL with the initial size of your cluster. etcd uses this URL to bootstrap your cluster and wire the new machines together. You can read the details on how this works by visiting https://discovery.etcd.io

$ curl https://discovery.etcd.io/new?size=3

After running this command you will get out a new private discovery URL that looks something like https://discovery.etcd.io/<token>. Take this token and put it in the relevant section of the user-data file using your favorite text editor. When you are done the top of the user-data file should look like this:

#cloud-config

coreos:
  etcd:
    discovery: https://discovery.etcd.io/<token>

Start the Cluster

By default, the CoreOS Vagrantfile will start a single machine. To start a cluster, we need to copy and modify the config.rb.sample file:

cp config.rb.sample config.rb

Uncomment the line containing num_instances and change it to 3 or more. Now we're ready to start the VMs:

$ vagrant up
Bringing machine 'core-01' up with 'virtualbox' provider...
Bringing machine 'core-02' up with 'virtualbox' provider...
Bringing machine 'core-03' up with 'virtualbox' provider...
==> core-01: Box 'coreos-alpha' could not be found. Attempting to find and install...
    core-01: Box Provider: virtualbox
    core-01: Box Version: >= 0
==> core-01: Adding box 'coreos-alpha' (v0) for provider: virtualbox
    core-01: Downloading: http://storage.core-os.net/coreos/amd64-usr/alpha/coreos_production_vagrant.box
    core-01: Progress: 46% (Rate: 6105k/s, Estimated time remaining: 0:00:16)

Vagrant will now download the latest CoreOS image and configure the VMs. You can ignore the warning about guest additions, everything will work just fine.

After the vagrant command returns it is time to ssh into one of your instances and try out a few commands. The only trick is that we will want to add the Vagrant key to our ssh-agent using ssh-add. This allows us to forward our SSH session to other machines in the cluster. You can do this with any key, such as those added via cloud-config, but our vagrant machines will already have the corresponding public key on disk.

$ ssh-add ~/.vagrant.d/insecure_private_key
Identity added: /Users/CoreOS/.vagrant.d/insecure_private_key (/Users/CoreOS/.vagrant.d/insecure_private_key)
$ vagrant ssh core-01 -- -A

Testing Out the Cluster

Now lets make sure that all of our machines came up and are set up with fleet.

$ fleetctl list-machines
MACHINE   IP            METADATA
517d1c7d... 172.17.8.101  -
cb35b356... 172.17.8.103  -
17040743... 172.17.8.102  -

And try setting a key into etcd which will be reflected on another machine in the cluster. We can run commands on other machines in the cluster by using fleetctl ssh and address them by the ID that is listed from list-machines. Be sure to replace the ID with an actual value from your list-machines:

$ etcdctl set first-etcd-key "Hello World"
Hello World
$ fleetctl ssh cb35b356 etcdctl get first-etcd-key
Hello World

There is a ton of functionality inside of fleet that won't be covered in this blog, but as our final task of trying out our new cluster let's schedule a job. Create a file called hello-fleet.service:

$ cat > hello-fleet.service
[Service]
ExecStart=/usr/bin/bash -c "while true; do echo 'Hello Fleet'; sleep 1; done"

Now tell fleet to start this service:

$ fleetctl start hello-fleet.service
Job hello-fleet.service scheduled to 517d1c7d.../172.17.8.101
$ fleetctl list-units
UNIT      LOAD  ACTIVE  SUB DESC  MACHINE
hello-fleet.service loaded  active  running - 517d1c7d.../172.17.8.101

Start Using Your Cluster

You now have a powerful little cluster on your laptop, complete with job scheduling, a distributed data store and a self-updating operating system. From here it is a choose-your-adventure of exploring fleet and etcd using the guides that we have put together.

And stay tuned for more blog posts that will outline how to use CoreOS and user-data to bootstrap clusters on other platforms.

etcd - The Road to 1.0

2014-04-14T00:00:00+00:00

Over the last few months the extent of community involvement and adoption of etcd has surpassed all our expectations. We wanted to take this opportunity to share with the wider community our plan for the ongoing development of etcd.

We want etcd to be a stable base for you to build distributed systems that are resilient to failures. Users of etcd should find things consistent and predictable, and client libraries should not need to be updated for years. In short, we need to get to 1.0, and this is clearly not where etcd is today.

We want to release etcd 1.0 later this year. This will require some significant changes which we have anticipated but, until now, specifically resisted the temptation to implement, so that we could maintain focus on working through complex, bigger picture problems.

Now that etcd has been baking in many production environments for some time, we’re ready to start forging ahead towards our 1.0 goal. At CoreOS, we ramped up the development team dedicated to the project. In the coming releases we will be focusing on:

Efficiency - reduce CPU, memory, and network usage
Scalability - run an arbitrary number of nodes in a cluster so nodes not participating in consensus can take over for downed peers
Reliability - we want etcd to run in many different types of environments while maintaining its guarantees
Testing - there is more to do here in terms of simulating network failures and fuzzing our system.

Next step: 0.4

Our upcoming release, etcd 0.4, adds a new feature, standby mode. It is an important first step in allowing etcd clusters to grow beyond just the peers that participate in consensus. This feature still needs work but we need your help testing it today. Additionally, we have bug fixes and other features that have landed. More details on this release when it ships soon.

If you have any questions, concerns, or want to participate in the roadmap of etcd 1.0, please get involved on github and join the mailing list where we will be posting regular, detailed updates.

GitHub:
https://github.com/coreos/etcd

Mailing List:
https://groups.google.com/forum/#!forum/etcd-dev

Cheers,
The etcd team @ CoreOS

Major Update: btrfs, docker 0.9, add users, writable /etc, and more!

2014-03-27T00:00:00+00:00

We locked the CoreTeam in a house for a week to make this release extra special. Here are all the new features:

New filesystem layout, writeable /etc/, read-only /usr/
Docker 0.9 (with btrfs support!)
btrfs stateful/root filesystem
Networking configured via networkd
CoreOS CloudInit, which enables boot time configuration of the following:
- Adding unix users (setting ssh keys, passwords, or import your ssh keys from github!)
- Write and control systemd units (start/stop, add a unit)
- Auto etcd bootstrapping on all platforms
- Write files to arbitrary locations
- Set up networking with networkd

To date, we've successfully pushed 38 versions of CoreOS out via our update service. Because of the filesystem switch, it's not possible to release these changes as an update. We will be providing new images for all platforms and have Amazon EC2 , Google Compute Engine, Openstack and Vagrant images available today.

These new images represent the first release on a path that ends at a stable, production-ready CoreOS.

New filesystem layout

The largest change was modifying our read-only root file system. In the current images, directories like /etc reside on the read-only partition because they contained parts of the OS that needed to stay in a known state in order to update them safely. We undertook a large effort to move all of these files into /usr, which now contains all of the critical files needed to operate the OS and is the only read-only portion of the filesystem. Every package that expected a hard-coded path to a file in /etc has been modified and moved into /usr. If you would like to provide your own version of a file that resided in /etc, you can now place it there and the OS will use it, instead of the one we provide by default.

With this change, systemd unit files now live in the standard location in /etc/systemd/system instead of within /media/state/units. When creating a unit, you can simply run systemctl enable foo.service.

Docker 0.9 and btrfs

CoreOS now supports Docker 0.9!

With the switch to a read-only /usr/, we also switched the rest of the filesystem to btrfs. This means you can now use Docker 0.9 with btrfs (which is much faster!), and any other native btrfs tools you might need (snapshotting, replication, etc). One unfortunate side effect of this change is that all Docker containers will need to be reinitialized on CoreOS (docker pull, or docker build). Docker does not provide any mechanisms to migrate between AUFS and btfs. We do not expect needing to make breaking changes like this again in the future.

As a result of a read/write /etc/, you can now easily modify docker.service to enable the docker remote api or tweak any other flags you wish. You can overwrite this file on disk manually or modify it on boot with cloud-config, as introduced below.

CoreOS CloudInit and cloud-config.yml

All CoreOS platforms going forward will use CoreOS CloudInit and the accompanying cloud-config.yml to configure the OS while it boots. Our goal is for you to be able to use the same cloud-config.yml to bootstrap CoreOS on any mix of platforms.

Example cloud-config.yml configuration file:

#cloud-config

coreos:
  etcd:
    # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
    # specify the intial size of your cluster with ?size=X
    discovery: https://discovery.etcd.io/<token>
    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
  units:
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start
    - name: my-container.service
      command: start
      content: |
        [Service]
        ExecStart=/usr/bin/docker run busybox /bin/sh -c "while true; do echo Hello World; sleep 1; done"
users:
  - name: core
    ssh-authorized-keys: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0g+ZTxC7weoIJLUafOgrm+h...
  - name: penny
    coreos-ssh-import-github: bcwaldon

This cloud-config.yml tells CloudInit to use do the following:

Use given etcd discovery token to auto-bootstrap the etcd cluster ($private_ip is automatically configured per platform, and swapped out with the real value)
Set the "core" unix users authorized_keys to the given key
Start CoreOS distributed fleet service
Create a systemd unit "my-container.service" and start it
Create the user "penny" and set the ~/.ssh/authorized_keys to the same public keys as the github user "bcwaldon"

The exact same cloud-config.yml can be specified on any platform that CoreOS supports. For example, on Amazon, you would paste it into your user-data. On OpenStack, you use config-drive, but can place the identical cloud-config.yml you pasted into Amazon user-data. CloudInit also used during our OEM/distribution process to configure networking with networkd and other settings per platform (depricating the /run convention).

Learn more about cloud-config.yml: Reference docs for all supported cloud-config.yml directives

The CoreOS alpha will still be a bit bumpy as we slim things down and make final tweaks to the alpha. However, the release puts us on the critical path to a beta, and a subsequent stable release in the next few weeks. We'll be updating all of the documentation over the next few days and release new images to support for the rest of our platforms.

Dynamic Docker links with an ambassador powered by etcd

2014-02-27T00:00:00+00:00

The ambassador pattern is a novel way to deploy sets of containers that are configured at runtime via the Docker Links feature.

In this proof of concept demo, we demonstrate deploying a redis instance that is registered with etcd. This demo is similar to the documented redis example for Docker links, except distributed across multiple hosts. A redis client will be linked with a redis server, regardless of what physical host either is hosted on, using a dynamic proxy that is reading the registration data off of etcd. The end goal is to be able to arbitrary deploy all of these containers using fleet and everything will be configured at runtime.

We will need the following set of containers to make all this happen.

Host A:

crosbymichael/redis - An off-the-shelf redis server.
polvi/docker-register - A docker/etcd registration container. This container will read off of the docker API and publish data to etcd.
polvi/simple-amb - This is a very simple ambassador that will forward traffic to the configured location passed via an arg. This is used to Link the docker registration container with etcd. This container could be removed on CoreOS, because etcd is at a known location, but is used for the purposes of demonstrating static versus dynamic ambassadors.

Host B:

polvi/dynamic-etcd-amb - This is where the magic happens. This is a dynamic proxy, powered by etcd, that watches a known etcd key and routes traffic to the container registered at the key. The key can change at runtime, and the proxy will update itself. If multiple instances are registered to the key space, the ambassador will start load balancing traffic to both containers.
relateiq/redis-cli - This is an off-the-shelf redis client for purposes of demonstrating the etcd powered ambassador.

The net result will look something like this:

Preparing for fleet

In order to deploy these set of containers, we will write a group of systemd service files that are fleet aware. This will allow us to deploy a whole set of containers and let them be scheduled arbitrary across the cluster of CoreOS hosts. These service files are re-usable on non-CoreOS systemd distros, assuming that docker is installed.

The easiest way to test this yourself is to spin up a CoreOS cluster on EC2 using our Cloud Formation "launch now" button.

Example service files

Below is a set of service files corresponding to the containers mentioned above, to be deployed via fleet, and supervised with systemd.

Host-A units

redis-demo.service

An off-the-shelf redis server.

[Service]
ExecStartPre=-/usr/bin/docker kill %n
ExecStartPre=-/usr/bin/docker rm %n
ExecStart=/bin/bash -c "HOST_IP=$(/bin/ifconfig eth0 | awk '/inet /{print $2}') && exec /usr/bin/docker run -rm -name %n -p $HOST_IP::6379 crosbymichael/redis"
ExecStop=/usr/bin/docker stop -t 3 %n

This container uses a bash trick to get the IP from eth0. We need this because we are going to look-up the IP/port combination and register it with etcd using the docker port command. By default, 0.0.0.0 will be registered if no IP is specified. This will not work as we need to know the network address of the host that is running the container.

Note: you will need to expose the 49000-50000 port range in your EC2 security group, if you are using our cloud formation.

We use the systemd %n variable to name the container the same as the systemd unit, in this case redis-demo.service. This is important for the registration of the container with etcd.

etcd-amb-redis.service

A simple ambassador needed to give our registration container access to etcd.

[Service]
ExecStartPre=-/usr/bin/docker kill %n
ExecStartPre=-/usr/bin/docker rm %n
ExecStartPre=-/usr/bin/docker pull polvi/simple-amb
ExecStart=/usr/bin/docker run -rm -name %n polvi/simple-amb 172.17.42.1:4001
ExecStop=/usr/bin/docker stop -t 3 %n

[X-Fleet]
X-ConditionMachineOf=redis-demo.service

polvi/simple-amb will forward all traffic it gets on port 10000 to the argument provided. In this case, 172.17.42.1:4001 is the known address for etcd on every CoreOS instance from inside of a Docker container-- we we simply statically forward all traffic there.

X-ConditionMachineOf tells fleet to schedule this systemd unit to the same machine as wherever redis-demo.service gets scheduled to. You can read more docs on the X-Fleet section over on the fleet docs.

redis-docker-reg.service

A container that reads port information off the docker API, and heartbeats it to etcd.

[Unit]
After=etcd-amb-redis.service

[Service]
ExecStart=/usr/bin/docker run -link etcd-amb-redis.service:etcd -v /var/run/docker.sock:/var/run/docker.sock -rm polvi/docker-register redis-demo.service 6379 redis-A

[X-Fleet]
X-ConditionMachineOf=redis-demo.service

This launches the polvi/docker-register container, as described above, that will read the IP and port information off of the docker API and publish that data to etcd under the service name of redis-A. There are two important aspect of this container.

It requires an etcd to publish to, so we -link in the simple ambassador pointing to etcd.
It talks to the host docker instance, so we use a docker volume to bind mount in the host docker.sock into the container. Note, this gives the container full control of the host dockerd! This is a security issue, but required for the container to read and then publish the port that was assigned by docker.

This unit also has a X-ConditionMachineOf to schedule it to the same machine as redis-demo.service. Finally, we use the After systemd directive to make sure the process is started after etcd-amb-redis.service (on the same machine) to make sure that it has a etcd to talk to when it is launched.

Host-B units

etcd-amb-redis2.service

A second simple etcd ambassador container, scheduled via fleet on a different host than redis-demo.service.

[Service]
ExecStartPre=-/usr/bin/docker kill %n
ExecStartPre=-/usr/bin/docker rm %n
ExecStart=/usr/bin/docker run -rm -name %n polvi/simple-amb 172.17.42.1:4001
ExecStop=/usr/bin/docker stop -t 3 %n

[X-Fleet]
X-Conflicts=redis-demo.service

This unit uses the fleet directive, X-Conflicts, to make sure that it gets scheduled to a host that is not the same as where redis-demo.service was scheduled, guaranteeing these containers will be on two different hosts.

redis-dyn-amb.service

The etcd powered dynamic ambassador!

[Unit]
After=etcd-amb-redis2.service

[Service]
ExecStartPre=-/usr/bin/docker kill %n
ExecStartPre=-/usr/bin/docker rm %n
ExecStart=/usr/bin/docker run -link etcd-amb-redis2.service:etcd -rm -name %n -p 127.0.0.1::6379 polvi/dynamic-etcd-amb redis-A 6379
ExecStop=/usr/bin/docker stop -t 3 %n

[X-Fleet]
X-ConditionMachineOf=etcd-amb-redis2.service

This tells the proxy to expose port 6379 and point it to the service registered as redis-A in etcd. X-ConditionMachineOf is used again to make it it gets deployed to where our second host matching wherever etcd-amb-redis2.service was deployed.

Deploying and testing with fleet

To deploy this with fleet, we simply run fleetctl start *.service in a directory containing all of these service files. All units will be scheduled across the cluster with our topology requirements in place.

Note: if you're deploying from your laptop, you'll need to use --tunnel, as specified by the fleet remote access docs.

To test that everything worked as expected, we will ssh to the host that is running the dynamic proxy and manually interact with it using a redis client container, relateiq/redis-cli, and a docker -link.

This command will ssh us to the host where that service is running.

fleetctl ssh -u redis-dyn-amb.service

From there, we launch a docker container with the redis client, on the shell:

$ docker run -i -t -link redis-dyn-amb.service:redis relateiq/redis-cli
redis 172.17.0.3:6379> ping
PONG

Success!

From here, you would use docker links environment variables to configure your application to point to the dynamic proxy.

A copy of these service files can be found at github.com/polvi/fleet-redis-demo.

Introduction to networkd, network management from systemd

2014-02-25T00:00:00+00:00

As of version 210, systemd now includes support for basic network configuration through udev and networkd. CoreOS sponsored the development of networkd. It will soon be available in CoreOS, so we would like to take the opportunity to introduce some of its features.

Configuring low-level network device settings

As an alternative to custom udev rules, udev now applies some common low-level configuration to network device as the devices appear. To specify the settings, drop .link files into /etc/systemd/network/. Let's have a look at an example and then go through the options:

[Match]
MACAddress=12:34:56:78:9a:bc
Driver=brcmsmac
Path=pci-0000:02:00.0-*
Type=wlan
Virtualization=no
Host=my-laptop
Architecture=x86-64

[Link]
Name=wireless0
MTUBytes=1450
BitsPerSecond=10M
WakeOnLan=magic
MACAddress=cb:a9:87:65:43:21

The [Match] section expresses that this file will match any wireless device with the given MAC address, using the brmsmac driver and connected to the given pci bus, when running on a non-virtualized 64bit x86 machine with hostname my-laptop. When such a device appears, the [Link] section says that udev will rename it to wireless0, change its MTU, speed and MAC address and enable Wake-On-Lan. All this happens before udev notifies the rest of userspace about the device, to avoid confusion due to, e.g., changing interface name and MAC address.

Two further keys deserve special mention: When MACAddressPolicy=persistent is given and the device has a random MAC address set by the kernel at boot, udev will reset it to one which is also random, but guaranteed to be the same for the given device between boots. Conversely, MACAddressPolicy=random instructs udev to always reset the MAC address to a different random one on each boot. Lastly, NamePolicy controls how the interface will be assigned a persistent name, if at all. It takes an ordered list of policies, and the first successful one is used to determine the name.

The default .link file is /usr/lib/systemd/network/99-default.link. It applies to all devices and only specifies MACAddressPolicy and NamePolicy.

Unlike udev rule files, only one .link file is applied to a given device. Namely the first one (in alphabetical order) that matches according to its [Match] section.

Setting up network connections

A new system daemon, networkd, supports setting up some basic (and some not so basic) network configurations. The support is mostly aimed at servers and containers, so it is a great fit for CoreOS.

networkd is configured using .network files, which work similarly to .link files described above. The same matching logic applies, with one additional option: namely to match on the interface name, using the Name key. It is worth noting that it is not safe to match on the names given by the kernel (eth0, eth1,...) as these may change between reboots. The only exception to warning is if you are matching on all of them with Name=eth*. Also, we should note that networkd is only made aware of new network devices after udev has applied its low-level settings to them, so we can safely match on the new MACAddress and Name that udev has set for our device.

Lets have a look at a basic .network file:

[Match]
Name=en*

[Network]
DHCP=yes

This will start DHCP on each interface whose name starts with en, which is the scheme used by udev's naming policies to name ethernet devices.

If you want to set up a static address on a specific device you could use:

[Match]
Name=enp2s0

[Network]
Address=192.168.0.15/24
Gateway=192.168.0.1

Lastly, if you want to set up the static address on enp2s0, and use DHCP for everything else, you would use the two above .network files, but make sure they are ordered correctly. E.g., name the first one 20-dhcp.network and the second 10-static.network, to make sure the static configuration matches enp2s0 before the dhcp one does. As for .link files, only the first matching .network file is applied.

Bridging, bonding and VLANs

The last feature of networkd I'd like to highlight is the support for creating and using virtual network devices.

Virtual network devices are created using .netdev files, which also support a limited form of the matching logic described above. More precisely, as there is no hardware to match on, they only support matching on their environment, i.e., Virtualization, Host and Architecture. This means that with the same configuration you can get different setups on different machines, or depending on whether or not we are running in a container, etc.

Bridge

Imagine you want to create a bridge and add your ethernet devices to it. You would then drop in a .netdev file:

[NetDev]
Name=bridge0
Kind=bridge

And set Bridge=bridge0 in the .network files applied to your devices. Under this setup, networkd will create bridge0 as soon as it starts, and add all your matching devices to it as they appear.

VLAN

Similarly, if you want to set up two VLAN's, you'd create two .netdev files:

[NetDev]
Name=vlan1
Kind=vlan

[VLAN]
Id=1

[NetDev]
Name=vlan2
Kind=vlan

[VLAN]
Id=2

And in your .network file you would set VLAN=vlan1 vlan2, to create the VLANs when your device appears.

Future plans

This is only the beginning! We are hard at work at adding more features, so any suggestions, feature requests or patches are very welcome. In the short-term we are looking forward to adding DHCP6 and IPv4LL support, introspection both via a library and via dbus, and runtime control via dbus and a commandline client.

Cluster-Level Container Deployment with fleet

2014-02-18T00:00:00+00:00

Today I'd like to introduce our latest project, fleet. fleet builds on etcd and systemd to provide a distributed, fault-tolerant platform for deploying applications on CoreOS clusters.

The Basics

With fleet, you can treat your CoreOS cluster as if it shared a single init system. It encourages users to write applications as small, ephemeral units that can easily migrate around a cluster of self-updating CoreOS machines.

Architecture

User job requests kick off a bidding process by the machines in the cluster. Based on the received bids the engine assigns the job to a machine. This process relies on two of etcd's key features: directory watch events and cluster-level locks. During the life of a job an event-driven architecture allows the cluster to react to membership changes and service state changes in real-time.

Read more about the architecture on GitHub.

So... what can it actually do?

Deploy docker containers on arbitrary hosts in a cluster
Distribute services across a cluster using machine-level anti-affinity
Maintain N instances of a service, re-scheduling on failure
Discover machines running in the cluster
Automatically SSH into the machine running a job

Demo

The API keys used in this video were revoked before it was posted — free AWS isn't a feature of CoreOS. Below are the systemd units used in the demo:

subgun-http.1.service

[Unit]
Description=subgun

[Service]
ExecStartPre=-/usr/bin/docker kill subgun-1
ExecStartPre=-/usr/bin/docker rm subgun-1
ExecStart=/usr/bin/docker run -rm -name subgun-1 -e SUBGUN_LISTEN=127.0.0.1:8080 -e SUBGUN_LISTS=recv@sandbox2398.mailgun.org -e SUBGUN_API_KEY=key-779ru4cibbnhfa1qp7a3apyvwkls7ny7 -p 8080:8080 coreos/subgun
ExecStop=/usr/bin/docker kill subgun-1

[X-Fleet]
Conflicts=subgun-http.*.service

GitHub Repo • Docker Index

subgun-presence.1.service

[Unit]
Description=subgun presence service
BindsTo=subgun-http.1.service

[Service]
ExecStartPre=-/usr/bin/docker kill subgun-presence-1
ExecStartPre=-/usr/bin/docker rm subgun-presence-1
ExecStart=/usr/bin/docker run -rm -name subgun-presence-1 -e AWS_ACCESS_KEY=AKIAIBC5MW3ONCW6J2XQ -e AWS_SECRET_KEY=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 -e AWS_REGION=us-east-1 -e ELB_NAME=bcwaldon-fleet-lb coreos/elb-presence
ExecStop=/usr/bin/docker kill subgun-presence-1

[X-Fleet]
MachineOf=subgun-http.1.service

GitHub Repo • Docker Index

Get up and running

fleet is available in CoreOS version 231.0.0 and newer. Get an etcd cluster up and running and simply call systemctl start fleet. You can then interact with your fleet cluster using the fleetctl command line tool available from GitHub. Binaries are available for both Linux and OS X.

Amazon

We have an AWS CloudFormation template that uses the new etcd cluster discovery protocol to provide autoscaled CoreOS clusters. Check it out on our EC2 docs.

Step 1 of a new CloudFormation stack.

Learn More

All documentation is located within the GitHub repo. Learn more about using the fleetctl client, deployment and configuration, system architecture and more.

etcd 0.3.0 - Improved Cluster Discovery, API Enhancements and Windows Support

2014-02-07T00:00:00+00:00

We're excited to release version 0.3.0 of etcd, a highly-available key value store for shared configuration and service discovery. Read on to learn about the changes in this release or hop over to GitHub to download the latest binaries.

New Features

Discovery API

The Discovery API automates the process of bootstrapping a cluster. In previous releases, a human had to pick a leader, pass a list of -peers to all of the followers and maintain that list going forward. The Discovery API takes a unique token as an input and fetches a list of peers that have already been registered with that token. If no peers have ever existed, the first peer to use that token will assume leadership of the cluster. Stale peers will be cleaned up automatically to allow you to use the same token as machines come and go.

Use the publicly-available discovery service at https://discovery.etcd.io, or run your own. All of the information you need is available in the Discovery API docs.

Other API Additions

This release does not break the existing v2 API but it does add two new features:

CompareAndDelete: etcd has let you create and update keys atomically but this release introduces atomic deletes too. This means that you can delete a key only if certain prerequisites like its index or value are correct.
prevNode: write requests will now return a prevNode field next to the existing node field. This entry represents the state of this key before the write was made and is useful for many uses like using a key as a simple atomic queue without a GET and PUT.

Improved Logging

We added an event-based logging system to get details about leader elections and cluster changes out of the core consensus system of etcd. This gives admins more clear and helpful information about the cluster state via log files.

go get and Windows Support

We recently moved to using goven for dependency management in etcd which means a few things:

go get github.com/coreos/etcd works and is a fast way to grab master
Go cross compilation works and etcd now has official Windows builds

Changes to Functionality

Goodbye info File

The info file was removed and is no longer used. This file was a cache of command line arguments and was causing a lot of confusion and bug reports. If you have an info file in your etcd data directory you can safely remove it.

Snapshots by Default

etcd uses a replicated log that is managed by the Raft consensus protocol. During normal usage, every write causes an entry to be appended to the log. In previous releases, the entire log was stored in memory, which caused memory usage to grow over the life of the cluster.

In this release we enabled -snapshot=true by default. This means that periodically etcd will flush older logs from memory to reduce resource usage.

Minor Fixes and Improvements

/mod/dashboard was refactored to improve the code quality
tracing options were added to enable use of pprof, graphite and other debugging methods
An etcd client matrix was added
Statistics APIs were documented

Bug Fixes

Improvements to the goraft locking code and leader step down code
Fixes to the internal watch datastructures and locks

Brandon's etcd presentation at GoSF

2014-01-16T00:00:00+00:00

Our very own @BrandonPhilips gave a talk at GoSF on etcd. Here are the slides.

Jumpers and the Software Defined Localhost

2014-01-13T00:00:00+00:00

The rise of Docker has been great thing for Linux containers and namespaces. Docker’s model of application building and deployment solves a lot of problems. However, one of the more common sticking points still being worked out is how dynamic, cross machine, container-to-container networking and service discovery will work.

We would like to propose a solution to this, called the "software defined localhost”.

Here is how it works, and it is pretty simple:

Containers hardcode a localhost port in their configuration (127.0.0.1:3306, for mysql, for example)
The container runtime presents a “jumper” (an open port) on the containers 127.0.0.1:3306, automatically and transparently connecting it to the configured remote container.

The net result is a system similar to synapse but on a container per container level.

We’re using the term “jumper” to refer to the tool that does this style of network namespace proxying.

For the sake of demonstration, we wrote a simple jumper called nspipe that will allow you to bind in one Linux network namespace, but do networking in another.

Terminal 1: Start a container and check for listening ports (currently none open)

$ docker run -t -i polvi/ubuntu-netstat /bin/bash
root@b202508907b0:/# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
root@b202508907b0:/#

Terminal 2: Start nspipe and attach a port on localhost:80 for b202508907b0

$ PID=$(./get-leader-pid b202508907b0)
$ sudo ./nspipe -t $PID -l 127.0.0.1:80 -r www.google.com:80
PROXY: targetPid:8231 targetAddr:localhost:80 remoteAddr:www.google.com:80

Terminal 1: Check if port is listening and test it

root@b202508907b0:/# netstat -lntup
Active Internet connections (only servers)
tcp        0      0 127.0.0.1:80            0.0.0.0:*               LISTEN      -   
root@b202508907b0:/# curl localhost:80
<HTML><HEAD><meta http-equiv="content-type” content="text/html;charset=utf-8">
...

Tada! We attached a port on localhost that forwards all traffic to google.com.

Note: ./get-leader-pid is a simple script that takes a docker container id and returns the first child process in the containers namespace. Using the setns system call requires directly poking the proc filesystem, and thus we need the pid.

A few more examples of this in action:

An etcd powered jumper that round-robin load balances to any services that are registered with etcd: https://github.com/coreos/nsproxy
A patched spiped (stunnel alternative) to securely encrypt all traffic that goes across the jumper, point to point. https://github.com/polvi/spiped

Finally, it's possible to setup the available networking before the container is started. In this example systemd-nspawn is a lightweight container manager that is bundled with systemd (and CoreOS), but it could be swapped out for the docker based lxc tools.

# create a fresh network namespace using iproute2
$ sudo ip netns add foo
# This sets up the jumper as a daemonized service via systemd
$ sudo systemd-run nspipe -p /var/run/netns/foo -l 127.0.0.1:80 -r www.google.com:80

$ sudo ip netns exec foo /usr/bin/systemd-nspawn -D busybox /bin/sh 
/ # ifconfig -a
lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B) 
/ # netstat -l
Active Internet connections (only servers)
tcp        0      0 127.0.0.1:80            0.0.0.0:*               LISTEN      -

A number of benefits are unlocked with this approach:

Networking is fully abstracted from the container, and always consistent (localhost always has what you need/specify). No dynamic IPs.
Application configuration can be static (just hardcode localhost:3306 for your DB, for example).
The jumper could transparently do things like dynamic service discovery, encrypting all traffic, starting other containers, without the container needing to implement anything.
Could follow the ambassador pattern for chaining services together, or just directly connect point to point, load balance, etc, let the platform decide.
The container could have no networking at all (only a loopback, no NAT), only given access to the services it needs (better security/isolation/consistency).
Convert the proxy to a load balancer, and you can start doing sophisticated things like rolling migration to a new service without needing to reconfigure existing clients.
Combined with systemd socket activation, the service receiving the connection on the other side of the proxy could be activated on demand. For example, you could bind 3306 into a container, and the mysql database container will provisioned on first connection.

Disadvantages:

All traffic needs to be ran through a proxy. At high load, this probably will not work. This could be overcome by having your application talk directly to whatever service discovery tool you would want to use, instead of having the runtime do it for you.
Implementing this requires directly fiddling with the kernel namespaces. This would have to be done natively in LXC to avoid races, or after a container is started with a helper tool similar in nature to pipework (but this would race at start-up.

We'd love to continue the discuss on coreos-dev and/or docker-user.

etcd 0.2.0 - new API, new modules and tons of improvements

2013-12-27T00:00:00+00:00

Today we released etcd 0.2.0 which introduces major features including an improved v2 API, two new modules for fair locking and leader election, and tons of new features.

Major Changes

v2 API

etcd 0.2.0 introduces a new API endpoint at /v2. This API has been built on all of the feedback we have gotten from the community of users who have been building projects on etcd. The major features of the new API include:

Simplified and consistent response format (example)
Compare-and-Swap (CAS) operations are now available so you can conditionally set values on keys (example)
Directories can be created and have a time-to-live (example)
Recursive GET requests to retrieve all data within a directory. (example)

The previous API from the 0.1 series of releases still works and can be found at /v1.

New Modules

One of the goals of etcd is to make building consistent and correct distributed systems easier for developers. In this release we have added two new modules that offer simple HTTP APIs to fair locks and leader elections.

The fair lock API can be used for things like ensuring only a single machine is running a cron job at a given time, for locking transactions across disparate data stores or as a building block for a larger distributed data structure. And you can do this from any language that can talk http:

On machine1 in your cluster you can protect job1:

curl -X POST 'http://127.0.0.1:4001/mod/v2/lock/job1?ttl=180?value=machine1'
# Do your work on job1
curl -X DELETE 'http://127.0.0.1:4001/mod/v2/lock/job1?value=machine1'

And on another machine you can protect job1:

curl -X POST 'http://127.0.0.1:4001/mod/v2/lock/job1?ttl=180&value=machine2'
# Do your work on job1 after machine1 has finished or timed out
curl -X DELETE 'http://127.0.0.1:4001/mod/v2/lock/job1?value=machine2'

Full docs for the lock are here.

The leader election module builds on top of the lock to provide a mechanism for doing a leader election on a set of candidate machines. We will do a blog post on how to wire this up to databases like Postgres for automatic failover. In the meantime you can find the docs here.

etcdctl improvements

etcdctl is a command line tool designed for use in scripts or for exploring the key space interactively and it has seen lots of improvements in this release.

exec-watch will execute a script everytime a key changes (example)
etcdctl ls can list directories recursively (example)
-o extended format for get/set for simple parsing from scripts (example)

Service Features

Configuration via command line, through environment variables, or by loading a configuration file.
Versioning of the internal protocol will allow you to run multiple versions of etcd in a cluster and perform rolling upgrades.
Bug fixes, internal refactoring and lots of tests added.

Getting Started

We will roll out 0.2.0 of etcd into CoreOS today. If you want to try it out in a Docker container or grab binary releases for Linux and OSX instruction are on the releases page.

Once you have etcd running the next step is reading through the extensive etcd manual and joining the mailing list or IRC channel.

Running etcd in Docker Containers

2013-12-13T00:00:00+00:00

Update: This blog post refers to functionality in an older version of etcd, 0.4.x. Check out the updated Docker guide for up-to-date information.

etcd is a highly-available key value store for shared configuration and service discovery. Running etcd within a docker container is a convenient way to deploy etcd or test out a sample cluster. Etcd containers are very easy to run — you just need to manipulate a few of the flags to make the networking work correctly. You can grab the etcd container or have docker download it automatically.

Start Containers

We're going to start three containers that use port 500x for the client communication and port 800x for the intra-cluster/raft communication. A few of the flags need to be modified per container:

-name needs to be unique
-peer-addr and -addr need to contain the IP where the machine can be reached on the outside of the container and a unique port number
-peers needs to contain a list of current members of the cluster and the port for server/raft communication (800x). The first container doesn't need the -peers list since it will be the master.

In this example, all three of these containers will be running on the same machine. Here are the three docker run commands that will get the cluster up and running. First, export a variable with your public IP address:

export PUBLIC_IP=54.196.167.255

And then start up our leader + two followers:

docker run -d -p 8001:8001 -p 5001:5001 quay.io/coreos/etcd:v0.4.6 -peer-addr ${PUBLIC_IP}:8001 -addr ${PUBLIC_IP}:5001 -name etcd-node1

docker run -d -p 8002:8002 -p 5002:5002 quay.io/coreos/etcd:v0.4.6 -peer-addr ${PUBLIC_IP}:8002 -addr ${PUBLIC_IP}:5002 -name etcd-node2 -peers ${PUBLIC_IP}:8001,${PUBLIC_IP}:8002,${PUBLIC_IP}:8003

docker run -d -p 8003:8003 -p 5003:5003 quay.io/coreos/etcd:v0.4.6 -peer-addr ${PUBLIC_IP}:8003 -addr ${PUBLIC_IP}:5003 -name etcd-node3 -peers ${PUBLIC_IP}:8001,${PUBLIC_IP}:8002,${PUBLIC_IP}:8003

Testing the Cluster

You should be able to test the cluster by curling any of the machines in the cluster or viewing the dashboard in a browser. Let's request /stats/leader which should indicate that all of our followers connected successfully:

curl -L 54.196.167.255:5001/v2/stats/leader

You should see something that looks like:

{"leader":"etcd-node1","followers":{"etcd-node2":{"latency":{"current":1.33342,"average":1.6575929235264224,"standardDeviation":0.8596793375097724,"minimum":1.034932,"maximum":27.428239},"counts":{"fail":4,"success":3936}},"etcd-node3":{"latency":{"current":1.448973,"average":1.7457679780163597,"standardDeviation":2.3964311316740465,"minimum":1.063433,"maximum":86.745084},"counts":{"fail":14,"success":3912}}}}

As you can see, etcd-node1 is the current leader and etcd-node2 and etcd-node3 are successfully connected as followers. You can check out the dashboard module at /mod/dashboard but you'll need to launch etcd with -cors='*' to make the cross-origin requests work.

Flags

Below is a quick description of the flags we used in this example and some other common ones:

Flag	Description
peers	Comma-separated list of cluster members. Any valid IP address will work. Use server/raft port for each member.
peers-file	A file containing the list of cluster members
name	The unique name of the node.
addr	Address that the etcd clients will connect to.
bind-addr	Interface that the etcd server will listen for client connections.
peer-addr	Address that the etcd server will advertise for intra-cluster/raft communication.
peer-bind-addr	Interface that the etcd server will listen for intra-cluster/raft connections.
data-dir	Path to the directory that stores data
ca-file	Path to a CA file. Used for SSL client certificates.
cert-file	Path to a certificate file. Used for SSL client certificates.
key-file	Path to a key file. Used for SSL client certificates.
peer-ca-file	Path to a CA file. Used for intra-cluster SSL communication.
peer-cert-file	Path to a certificate file. Used for intra-cluster SSL communication.
peer-key-file	Path to a key file. Used for intra-cluster SSL communication.
cors	White-listed addresses that cross-origin requests are allowed from .

CoreOS alpha updates

2013-12-09T00:00:00+00:00

Just a few updates on the CoreOS alpha for you. The latest release (160.0.1) includes:

Docker 0.7.1 went out with todays update, so you should already have it!
Support for Rackspace
Support for Google Compute Engine
We posted a short tutorial for setting up an IRC bouncer on CoreOS
We’ve added Infiniband and RealTek ethernet driver support to our kernel

Lastly, we have a few big changes to CoreOS and etcd that we’d like to make sure everyone knows is coming down the pipeline:

We intend tweak the read-only root filesystem to only be read-only on /usr/. This will allow you to overwrite configs by placing them on the filesystem where you might expect.
etcd 0.2.0 is nearly complete, including a new API. If you have not already, please be testing with etcd master on github. We are ironing out the major kinks, but consider etcd 0.2.0 the way of the future. We will be shipping it inside of CoreOS in the coming weeks.

Running a Utility Cluster on CoreOS

2013-12-04T00:00:00+00:00

CoreOS is designed to deliver high-density compute resources. The best way to take advantage of this concept on a small scale is run a personal utility cluster (or single machine) to run various utilities like IRC bouncers and the handful of websites that you inevitably end up hosting. CoreOS is a small, efficient operating system, which means you get more bang for your buck even on small cloud instances.

I'm going to walk through setting up ZNC, an IRC bouncer, on a CoreOS cluster. The basic process is to boot a CoreOS server, download a pre-made ZNC container and run the container via systemd.

Booting Your CoreOS Machine(s)

CoreOS runs on most cloud providers — Amazon EC2, Rackspace, Google Compute Engine and more — so pick your favorite and follow the very short setup process. If you have any extra physical servers laying around, CoreOS can also be booted via PXE or virtualized under KVM/QEMU, VMware and more.

You must login to a CoreOS machine with SSH keys — passwords are not supported at all. We're only supporting good habits! To connect run:

ssh core@your_ip_address

Set up the ZNC Container

Let's pull down the container onto the CoreOS machine. It's worth noting that docker will do this for you if it can't find the image locally, but we're including it explicitly so you can understand what's going on.

docker pull coreos/znc

Now we need to run through the intial set up process provided by ZNC. To do this, we need to run a bash prompt inside the container:

docker run -t -i coreos/znc /bin/bash

After the container starts, we need to run the setup command. Since ZNC doesn't allow you to run it as root, we need to specify a different user (irc)to run the command with su:

su - irc -c 'znc --makeconf'

Follow the steps when prompted to completely set up ZNC. I suggest listening on port 6000, don't use SSL and enabling all of the default modules. Refer to ZNC's configuration guide if you run into any issues. When it completes, we need to exit to stop the container so we can commit our changes.

Commit the Container Changes

Grab the container ID from docker ps -a (at the top of the list) and insert it into:

docker commit container_id yourusername/znc

You're customized ZNC container is now saved locally on your CoreOS machine. Now we need to run it. This time we need to forward the port you selected into the container. In my examples I'll be using 6000. Instead of starting ZNC from our bash prompt, we can start it directly by telling docker to run ZNC in the foreground. Running in the foreground is important because our container will stop running when the command it runs is completed.

docker run -d -p 6000:6000 yourusername/znc su - irc -c 'znc --foreground'

Connect via IRC Client

Connect to your ZNC server by using the public IP address, port and other configuration options you set up. You may have to debug as necessary.

Write the Unit File

CoreOS uses systemd to manage containers that get run by the operating system. For your ZNC container to restart after an automatic update, we need to write a unit file to manage it. These files are pretty straightforward and you can read about more complex examples in our Getting Started with systemd guide.

[Unit]
Description=ZNC IRC Service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker run --rm -p 6000:6000 yourusername/znc su - znc -c 'znc --foreground'

[Install]
WantedBy=local.target

The --rm flag will remove old copies of this container when they stop. This prevents the disk from filling up with containers that we don't want to ever run again.

Next Steps

You should now have a fully set up utility cluster (or server) running a single container. Next steps would include setting up a simple static site hosted within a container or construct your own container that runs another utility. Check out our guides for Getting Started with docker and Getting Started with etcd.

CoreOS on Google Compute Engine

2013-12-02T00:00:00+00:00

CoreOS can now be run on Google Compute Engine! We're working hard to support all of the awesome unique features of GCE, but for now CoreOS functions the same as our other supported platforms.

You can find docs on how to get up and running on Compute Engine on our documentation site and then follow our CoreOS Quickstart Guide.

If you're interested in more, we have guides that dive deeper into etcd, docker, and systemd.

etcd v0.1.2 with a new dashboard and bugfixes

2013-10-10T00:00:00+00:00

This etcd release, v0.1.2, is putting some final polish on the 0.1 series of releases. The big feature is a new dashboard designed to visualize the latency of the followers in the network and give you the ability to browse and edit the keyspace. Checkout this Youtube video for details.

Latency Chart

This chart shows the latency of the cluster from the master's point of view

Keyspace Browser

The key space browser lets you explore, modify, add and delete keys

Changelog

And here is the the full list of changes:

New dashboard available at /etcd/mod/dashboard
Introduced /v1/stats/leader and /v1/stats/self endpoints
/v1/keys accepts PUT and POST
New community created libraries for Clojure, Java, Python, C
cross-origin resource sharing whitelist via the -cors flag
Systemd journal support
Several bug fixes related to timeouts, testAndSet, and more
Powershell build scripts were added
Instructions on contributing to etcd added to CONTRIBUTING.md

v2 API Pre-Release

We will continue to merge bug fixes into the master branch for the 0.1 series. Most development work has shifted towards the 0.2 branch as we finalize the big pieces needed for the v2 API. This new API adds atomic sets based on index, improved handling of directories, and plenty of other fixes based on the communities feedback. We will have a pre-release version out soon so you can start working against the new API to make sure we get it right. All of the old clients should work too in the /v1/ namespace.

Getting Started

As always trying out etcd is as easy as downloading one of the binaries below and extracting it and then running:

./etcd
./etcdctl set hello-world "This is my first key"
./etcdctl get hello-world

Binary releases: OSX 64-bit, Linux amd64

GitHub repo: github.com/coreos/etcd

Thank you to all of the contributors in this release:

Andrew Hobden, AndyPook, Antonio Terreno, Brandon Philips, David Fisher, Deniz Adrian, Derek Chiang (Enchi Jiang), Diwaker Gupta, Evan, Fabrizio (Misto) Milo, Fatih Arslan, Geoff Hayes, Yifan Gu, Jose Plana, Michael Burns, Michael Marineau, Michael Stillwell, Rob Szumski, Roberto Aguilar, Theo Hultberg, Xiang Li, kelseyhightower

Boot on Bare Metal with PXE

2013-09-11T00:00:00+00:00

There have been a lot of requests from all of you about running CoreOS on physical gear. You want to use all of the features of CoreOS like atomic system upgrades, distributed configuration and Docker containers with the increased speed, flexibility and simplicity of bare metal. It makes a lot of sense to us too and we have been working to get the pieces in place for you to get CoreOS on your bare metal.

Run from RAM

This first version of bare metal CoreOS boots over PXE. If you aren't familiar PXE is a way to do network boots and is a common way to bring up installers.

But, we didn't want to force people to install CoreOS on disk and by default the PXE image runs completely out of RAM. This means you can use all of the features of CoreOS with zero on-disk installation. Plus, the image takes an ssh public key on the commandline passed from the PXE server so that you can log right in using your existing keypair.

The process simply becomes: boot up a machine, ssh in, and start running containers without touching storage.

# pxe boot and wait
ssh core@10.0.2.15
docker run -i -t busybox /bin/sh

To get started with this new feature checkout our guide on PXE Booting CoreOS.

Or Install to Disk

Of course if you want to have data persistant over reboots that is easily done. You can configure CoreOS to use a partition as it's STATE partition. The CoreOS image and root filesystem comes from PXE still but all of your data stays on disk.

mkfs.ext4 -L STATE /dev/sda1

Or even install CoreOS completely to disk. This will make everything on bare metal behave similarly to the current VM images.

coreos-install -d /dev/sda -V 72.0.0

Hardware Support

Give it a shot, but we are still working on the full set of hardware that we will be supporting. We have most of the common hardware working. If you run into issues ping on us #coreos or email Carly.

OpenStack, VMware and KVM images available

2013-08-28T00:00:00+00:00

We just uploaded images for OpenStack, VMware and KVM to make it easy for more of you to be involved in the CoreOS developer alpha. These images come fresh from our build system and we would love your feedback on what is working and what isn't.

VMware

We have been testing under VMware Fusion but these images should work under other VMware products too. If you give it a go on ESXi or another VMware platform let us know! The docs for this image are available over here.

A new Vagrant box with VMware support has been uploaded too. If you are using Hashicorp's VMware provider you can enjoy the speed and stability of vmware+vagrant by following our updated vagrant docs.

QEMU/KVM

QEMU/KVM is Linux's workhorse full virtualization tool and is used on desktops and servers. CoreOS has been booting on QEM/KVM since the start but the production images we uploaded today includes tools to easily add your ssh public keys without mounting the image and futzing around with the filesystem. To get started with KVM checkout the docs here.

OpenStack

OpenStack is an open source cloud platform that can be used on top of a variety of hypervisors. It has alot of flexibililty and we might have missed things in this first release. So, boot CoreOS up and let us know if we support your stack. Details and links in the OpenStack Guide.

Feedback

We want to make it easy for every developer to get involved in the CoreOS alpha and we will be making more images available soon. If the platform you want CoreOS on isn't currently supported let us know!

And if you have feedback or questions about these new image types join our IRC channel or send an email to the mailing list.

etcd v0.1.0 release

2013-08-11T00:00:00+00:00

etcd is ready for its first release! We just marked v0.1.0 and uploaded binaries for Linux and OSX.

These are the major features in this release:

Simple: curl'able user facing API (HTTP+JSON)
Secure: optional SSL client cert authentication
Fast: benchmarked 1000s of writes/s per instance
Reliable: properly distributed using Raft
Persistent: cluster failure is recoverable from disk

Check out the full etcd guide for details on what etcd is, how to setup clustering, configuring client cert authentication and more.

Community

A number of people in the etcd community have helped out with this release: Several bugs were fixed by Fabrizio Milo, Cong Ding and many others others. Set commands were added to etcdctl by Emil Stenqvist.

etcd projects

A number of independent etcd projects have been started as well: A dynamic reverse proxy written in Go and built on top of the Go etcd client library was started by calavera. Three Ruby bindings have been created by iconara, jpfuentes2 and ranjib. And a Chef cookbook was written by spheromak.

Onward to v0.2.0

We are just getting started with this release and have started planning for v0.2.0. Some of the features we want to build include: TTLs on entire directories, improved log snapshot scheduling and client access control. Kick the tires on this release and join the discussion about the next version.

CoreOS Vagrant Images

2013-08-02T00:00:00+00:00

Our first vagrant images are ready to try. If you are new to vagrant checkout our vagrant guide for install instructions. Already have vagrant? Then it is as simple as:

git clone https://github.com/coreos/coreos-vagrant/
cd coreos-vagrant
vagrant up
vagrant ssh

Thanks to @mitchellh (vagrant creator) for helping us get this going.

Please give it a shot and let us know how it goes! If you run into a bug, please file it in the github repo. Here are some getting started tips.

We think it'd be really cool to have multiple vagrant instances performing service disovery together via etcd (think: your own docker cloud distributed across laptops!). If this is something you'd want to help figure out, jump in #coreos on freenode!

Up next: We're testing CoreOS with a few companies on their physical servers. If that is interesting to you, please send me the output of these commands from servers that you'd like to test:

udevadm info --export-db
lspci -vv

Update: This only works on VirtualBox at the moment. We will add VMware soon.

Distributed configuration data with etcd

2013-07-23T00:00:00+00:00

Today we would like to announce etcd, a highly available key value store for shared configuration data. CoreOS built etcd to solve the problem of shared configuration and service discovery. etcd is inspired by projects like Zookeeper, or doozer, but is a completely new piece of software. Some of the key design features:

Simple: curl'able user facing API (HTTP+JSON)
Secure: optional SSL client cert authentication
Fast: benchmarked 1000s of writes/s per instance
Reliable: Properly distributed and replicated log using raft
Open source: Active, well maintained, open source project

etcd uses the raft algorithm; a consensus algorithm for replicated logs. The CoreOS team has been heavy contributors to the go-raft implementation, which is maintained by benbjohnson. We have our own implementation, but it is mainly a staging repo before being merged upstream. Thank you to Ben for his awesome contributions to date!

etcd is written in Go.

Check out the documentation on github for many examples of how to use etcd to share configuration data, watching for changes on particular keys, or perform master election.

Recoverable System Upgrades

2013-07-16T00:00:00+00:00

System upgrades can introduce problems, and when upgrades go bad, manual steps need to be taken to get machines back up. The problem is compounded on public cloud infrastructure since you often have large numbers of machines, and getting to a recovery console can take a while.

We believe in tight release cycles. CoreOS is designed to keep your servers secure against the latest exploits and to give you access to the latest features and fixes in weeks or months, not years. So, how do we balance tight release cycles with a need for stability? We built some systems and technology into CoreOS to help balance it all out.

Enabling consistency and rollback

Upgrading CoreOS is a bit different than the usual distros. Our update system is based on ChromeOS. The big difference is that we have two root partitions, which we'll call root A and root B. Initially, your system is booted into the root A partition, and CoreOS begins talking to the update service to find out about new updates. If there is an update available it is downloaded and installed to root B. And to ensure we don't disrupt your application we rate limit the disk and network I/O this process is allowed to use by using Linux cgroups.

Using this dual-root scheme is an improvement on the existing workflow of yum or apt-get. Often when upgrading using these tools you can find that the package manager will kick daemons to get them using new libraries or move configuration files around. And because of that you should treat system upgrades as a deploy: taking machines out of load balancers and clusters.

But, on CoreOS, since the root partition your application is running on top of isn't modified in place, your server is never in an unstable or partially upgraded state. To complete the upgrade, simply reboot the machine, and in a few seconds you will start running on root B with a freshly updated system.

What happens when things go wrong?

To further protect against a bad upgrade, CoreOS has an automated system for rollback. If a machine fails to come up after an upgrade, simply reboot the machine and it will revert to the previous root partition.

This works by setting some metadata on the partition table. When root B gets its upgrade it has a "tries" counter that is set to 1, telling our boot system that we should attempt to use this root once, and if it fails to come up, then on the next boot revert to using root A.

The end result is that CoreOS gives you frequent, consistent updates to the core system that your applications rely on, while giving you automated tools to reduce the risks involved.