Docs, polish, clean-ups for KMS and health-check features

6 files changed, 217 insertions, 38 deletions

diff --git a/README.md b/README.md
index ac9343e..2433e76 100644
--- a/README.md
+++ b/README.md
@@ -22,7 +22,7 @@ Requires latest stable Rust to compile. Contributions welcome, see
 
 ## Building and Running
 
-Requires the latest stable Rust to build.
+Requires Rust 1.28 or newer to build.
 
 ```bash
 # Build roughenough
@@ -33,7 +33,7 @@ The client binary is `target/release/roughenough-client`. After building you can
 binary and run on its own (no `cargo` needed) if you wish.
 
 ```bash
-$ cp target/release/roughenough-server /usr/local/bin 
+$ cp target/release/roughenough-client /usr/local/bin 
 ```
 
 ### Using the Client to Query a Roughtime Server 
@@ -41,7 +41,7 @@ $ cp target/release/roughenough-server /usr/local/bin
 ```bash
 $ target/release/roughenough-client roughtime.int08h.com 2002
 Requesting time from: "roughtime.int08h.com":2002
-Received time from server: midpoint="Jul 28 2018 15:21:31", radius=1000000 (merkle_index=0, verified=false)
+Received time from server: midpoint="Oct 26 2018 23:20:44", radius=1000000, verified=No (merkle_index=0)
 ```
 
 ### Validating Server Responses 
@@ -56,10 +56,10 @@ roughtime.int08h.com descriptive text "016e6e0284d24c37c6e4d7d8d5b4e1d3c1949ceaa
 # Validate the server response using its public key
 $ target/release/roughenough-client roughtime.int08h.com 2002 -p 016e6e0284d24c37c6e4d7d8d5b4e1d3c1949ceaa545bf875616c9dce0c9bec1
 Requesting time from: "roughtime.int08h.com":2002
-Received time from server: midpoint="Jul 28 2018 15:26:54", radius=1000000 (merkle_index=0, verified=true)
+Received time from server: midpoint="Oct 26 2018 23:22:20", radius=1000000, verified=Yes (merkle_index=0)
 ```
 
-The **`verified=true`** in the output confirms that the server's response had a valid signature.
+The **`verified=Yes`** in the output confirms that the server's response had a valid signature.
 
 ### Server Configuration
 
@@ -74,9 +74,11 @@ YAML Key | Environment Variable | Necessity | Description
 --- | --- | --- | ---
 `interface` | `ROUGHENOUGH_INTERFACE` | Required | IP address or interface name for listening to client requests
 `port` | `ROUGHENOUGH_PORT` | Required | UDP port to listen for requests
-`seed` | `ROUGHENOUGH_SEED` | Required | A 32-byte hexadecimal value used to generate the server's long-term key pair. **This is a secret value and must be un-guessable**, treat it with care.
+`seed` | `ROUGHENOUGH_SEED` | Required | A 32-byte hexadecimal value used to generate the server's long-term key pair. **This is a secret value and must be un-guessable**, treat it with care. (If compiled with KMS support, length will vary; see [Optional Features](#optional-features))
 `batch_size` | `ROUGHENOUGH_BATCH_SIZE` | Optional | The maximum number of requests to process in one batch. All nonces in a batch are used to build a Merkle tree, the root of which is signed. Default is `64` requests per batch.
 `status_interval` | `ROUGHENOUGH_STATUS_INTERVAL` | Optional | Number of _seconds_ between each logged status update. Default is `600` seconds (10 minutes).
+`health_check_port` | `ROUGHENOUGH_HEALTH_CHECK_PORT` | Optional | If present, enable an HTTP health check responder on the provided port. Be careful with this, see [Optional Features](#optional-features).
+`key_protection` | `ROUGHENOUGH_KEY_PROTECTION` | Optional | If compiled with KMS support, the ID of the KMS key used to protect the long-term identity. See [Optional Features](#optional-features).
 
 #### YAML Configuration 
 
@@ -110,6 +112,7 @@ $ /path/to/roughenough-server ENV
 ### Starting the Server
 
 ```bash
+# Build roughenough
 $ cargo build --release
 
 # Via a config file
@@ -141,6 +144,19 @@ $ cp target/release/roughenough-server /usr/local/bin
 
 Use Ctrl-C or `kill` the process.
 
+
+## Optional Features
+
+Roughenough has opt-in features enabled either A) via a config setting, or B) at compile-time.
+
+* A simple HTTP health-check responder to facilitate detection and replacement
+  of "sick" Roughenough servers.
+* Use of encryption and cloud Key Management Systems (KMS) to protect the 
+  long-term server identity.
+
+See [OPTIONAL-FEATURES.md](doc/OPTIONAL-FEATURES.md) for details.
+
+
 ## Limitations
 
 Roughtime features not implemented by the server:
@@ -152,11 +168,6 @@ Roughtime features not implemented by the server:
   smeared leap-seconds but time sourced from members of `pool.ntp.org` likely will not.
 * Ecosystem-style response fault injection.
 
-Other notes:
-
-* Per-request heap allocations could probably be reduced: a few `Vec`'s could be replaced by 
-  lifetime scoped slices.
-
 ## About the Roughtime Protocol
 [Roughtime](https://roughtime.googlesource.com/roughtime) is a protocol that aims to achieve rough 
 time synchronisation in a secure way that doesn't depend on any particular time server, and in such
diff --git a/doc/OPTIONAL-FEATURES.md b/doc/OPTIONAL-FEATURES.md
new file mode 100644
index 0000000..babf1f0
--- /dev/null
+++ b/doc/OPTIONAL-FEATURES.md
@@ -0,0 +1,168 @@
+# Optional Features
+
+These are opt-in features enabled either A) via a config setting, or B) at compile-time.
+
+* [An HTTP Health Check responder](#http-health-check)
+* [Key Management System (KMS) support](#key-management-system-kms-support)
+
+
+# HTTP Health Check
+
+## Description 
+
+Intended for use by load balancers or other control plane facilities to monitor 
+the state of Roughenough servers and remove unhealthy instances automatically. 
+
+The server unconditionally emits a response to any TCP connection to the health
+check port then closes the connection:
+
+```http
+HTTP/1.1 200 OK
+Content-Length: 0
+Connection: Close
+
+```
+
+No attempt is made to parse the request, the server *always* emits this response. 
+
+# How to enable
+
+Provide a value for the `health_check_port` setting. This enables the simplistic 
+HTTP health check responder on the configured port. 
+
+## Warning
+
+**An unprotected health-check port can be used to DoS the server. Do NOT expose 
+the health check port to the internet!** 
+
+To accurately reflect the ability of a Roughenough server to respond to requests, 
+the health check socket is serviced in the same event loop executing the primary Roughtime 
+protocol. Abuse of the health-check port can denial-of-service the whole server.
+
+If enabled, ensure the health check port is accessible only to the *intended load-balancer(s)
+and/or control plane components*.
+
+
+# Key Management System (KMS) Support
+
+## Description 
+
+The server's long-term identity can be protected by encrypting it, storing the encrypted value
+in the configuration, and invoking a cloud key management system to temporarily decrypt 
+(in memory) the long-term identity at server start-up. 
+
+This way the server's long-term identity is never stored in plaintext. Instead the encrypted 
+long-term identity "blob" is safe to store on disk, on Github, in a container, etc. Ability 
+to access the unencrypted identity is controlled "out of band" by the KMS system.
+
+## How to enable KMS support
+
+KMS support must be compiled-in. To enable:
+
+```bash
+# Build with Google Cloud KMS support
+$ cargo build --release --features "gcpkms"
+
+# Build with AWS KMS support
+$ cargo build --release --features "awskms"
+```
+
+## Google or Amazon: choose one and one only
+
+Sadly, due to incompatibilities with dependencies of the KMS libraries, only **one** 
+KMS system can be enabled at a time. Attempting `--features "awskms,gcpkms"` will result
+in a build failure.
+
+## Using `roughtime-kms` to encrypt the long-term seed
+
+Use the command line tool `roughtime-kms` to encrypt the seed value for the 
+server's long-term identity. To do this you will need: 
+
+ 1. The long-term key seed value 
+ 2. Access credentials for your cloud of choice
+ 3. An identifier for the KMS key to be used
+ 4. Necessary permissions to perform symmetric encrypt/decrypt operations
+    using the selected key
+
+For Amazon the key identifier is an ARN in the form:
+```
+arn:aws:kms:SOME_AWS_REGION:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
+```
+
+For Google the key identifier is a resource ID in the form:
+```
+projects/PROJECT_NAME/locations/GCP_LOCATION/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME
+```
+
+### AWS Example
+
+#### Credentials 
+
+Mercifully, the Rusoto library handles AWS credentials pretty smoothly as described in 
+[Rusoto's documentation](https://github.com/rusoto/rusoto/blob/master/AWS-CREDENTIALS.md). 
+
+#### Command line
+
+```bash
+# Build roughenough with AWS KMS support
+$ cargo build --release --features "awskms"
+
+# Encrypt the seed value
+$ target/release/roughenough-kms \
+  -k arn:aws:kms:SOME_AWS_REGION:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab \
+  -s a0a31d76900080c3cdc42fe69de8dd0086d6b54de7814004befd0b9c4447757e
+  
+# Output of above will be something like this
+key_protection: "arn:aws:kms:SOME_AWS_REGION:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
+seed: b8000c000102020078d39e85c7386e9e2bed1f30fac6dd322db96b8aaac8974fc6c0e0f566f8f6c971012fca1e69fffffd947fe82a9e505baf580000007e307c06092a864886f70d010706a06f306d020100306806092a864886f70d010701301e060960864801650304012e3011040c55d16d891b3b2a1ae2587a9c020110803bcc74dd96336009087772b28ec908c40e4113b1ab9b98934bd3b4f3dd3c1e8cdc6da82a4321fd8378ad0e2e0507bf0c5ea0e28d447e5f8482533baa423b7af8459ae87736f381d87fe38c21a805fae1c25c43d59200f42cae0d07f741e787a04c0ad72774942dddf818be0767e4963fe5a810f734a0125c
+```
+
+Copy and paste the output `key_protection` and `seed` values into a config or
+set the corresponding environment variables. `roughenough-server` will detect that
+AWS KMS is being used and decrypt the seed automatically.
+
+### GCP Example
+
+#### Credentials
+
+Only **Service Account credentials** (in `.json` format) are currently supported. OAuth, bearer tokens, 
+GAE default credentials, and GCE default credentials are **not** supported (contributions to
+add support are particularly welcome!).
+
+To get Service Account credentials:
+
+1. Creating a new service account?
+    1. Create the account 
+    2. Download the credentials when prompted
+2. Existing service account?
+    1. Open the Cloud Console https://console.cloud.google.com
+    2. Navigate to `IAM -> Service accounts`
+    3. Locate the service account row, click on its "Actions" menu (the three dots on the right) 
+    4. Choose `Create key` and `JSON` format
+    5. Download the credentials when prompted
+
+Make note of the full path where the credentials are saved, it's needed in the next step.
+
+#### Command line
+
+```bash
+# Set environment variable pointing to downloaded Service Account credentials
+$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json 
+
+# Build roughenough with Google KMS support
+$ cargo build --release --features "gcpkms"
+
+# Encrypt the seed value
+$ target/release/roughenough-kms \
+  -k projects/PROJECT_NAME/locations/GCP_LOCATION/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME \
+  -s a0a31d76900080c3cdc42fe69de8dd0086d6b54de7814004befd0b9c4447757e
+  
+# Output of above will be something like this
+key_protection: "projects/PROJECT_NAME/locations/GCP_LOCATION/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME"
+seed: 71000c000a2400c7f2553954873ef29aeb37384c25d7a937d389221207c3368657870129d601d084c8da1249008d6fd4640f815596788e97bb3ce02fd007bc25a1019ca51945c3b99283d3945baacd77b1b991f5f6f8848c549a5767f57c9c999e97fe6d28fdb17db1d63c2ea966d8236d20c71e8e9c757c5bab62472c65b48376bc8951700aceb22545fce58d77e7cc147f7134da7a2cca790b54f29e4798442cee6e0d34e57f80ce983f7e5928cceff2
+```
+
+Copy and paste the output `key_protection` and `seed` values into a config or
+set the corresponding environment variables. `roughenough-server` will detect that
+Google KMS is being used and decrypt the seed automatically.
+
diff --git a/src/bin/roughenough-client.rs b/src/bin/roughenough-client.rs
index 9318417..5d7c25c 100644
--- a/src/bin/roughenough-client.rs
+++ b/src/bin/roughenough-client.rs
@@ -36,7 +36,7 @@ use std::net::{ToSocketAddrs, UdpSocket};
 use clap::{App, Arg};
 use roughenough::merkle::root_from_paths;
 use roughenough::sign::Verifier;
-use roughenough::{RtMessage, Tag, CERTIFICATE_CONTEXT, SIGNED_RESPONSE_CONTEXT, VERSION};
+use roughenough::{RtMessage, Tag, CERTIFICATE_CONTEXT, SIGNED_RESPONSE_CONTEXT, roughenough_version};
 
 fn create_nonce() -> [u8; 64] {
     let rng = rand::SystemRandom::new();
@@ -136,7 +136,7 @@ impl ResponseHandler {
                 &self.cert[&Tag::SIG],
                 &full_cert
             ),
-            "Invalid signature on DELE tag!"
+            "Invalid signature on DELE tag, response may not be authentic"
         );
     }
 
@@ -146,7 +146,7 @@ impl ResponseHandler {
 
         assert!(
             self.validate_sig(&self.dele[&Tag::PUBK], &self.msg[&Tag::SIG], &full_srep),
-            "Invalid signature on SREP tag!"
+            "Invalid signature on SREP tag, response may not be authentic"
         );
     }
 
@@ -162,7 +162,7 @@ impl ResponseHandler {
 
         let hash = root_from_paths(index as usize, &self.nonce, paths);
 
-        assert_eq!(hash, srep[&Tag::ROOT], "Nonce not in merkle tree!");
+        assert_eq!(hash, srep[&Tag::ROOT], "Nonce is not present in the response's merkle tree");
     }
 
     fn validate_midpoint(&self, midpoint: u64) {
@@ -177,12 +177,12 @@ impl ResponseHandler {
 
         assert!(
             midpoint >= mint,
-            "Response midpoint {} lies before delegation span ({}, {})",
+            "Response midpoint {} lies *before* delegation span ({}, {})",
             midpoint, mint, maxt
         );
         assert!(
             midpoint <= maxt,
-            "Response midpoint {} lies after delegation span ({}, {})",
+            "Response midpoint {} lies *after* delegation span ({}, {})",
             midpoint, mint, maxt
         );
     }
@@ -196,7 +196,7 @@ impl ResponseHandler {
 
 fn main() {
     let matches = App::new("roughenough client")
-    .version(VERSION)
+    .version(roughenough_version().as_ref())
     .arg(Arg::with_name("host")
       .required(true)
       .help("The Roughtime server to connect to")
@@ -309,10 +309,11 @@ fn main() {
         let nsecs = (midpoint - (seconds * 10_u64.pow(6))) * 10_u64.pow(3);
         let spec = Utc.timestamp(seconds as i64, nsecs as u32);
         let out = spec.format(time_format).to_string();
+        let verify_str = if verified { "Yes" } else { "No" };
 
         println!(
-            "Received time from server: midpoint={:?}, radius={:?} (merkle_index={}, verified={})",
-            out, radius, index, verified
+            "Received time from server: midpoint={:?}, radius={:?}, verified={} (merkle_index={})",
+            out, radius, verify_str, index
         );
     }
 }
diff --git a/src/bin/roughenough-kms.rs b/src/bin/roughenough-kms.rs
index 389d076..b9099cd 100644
--- a/src/bin/roughenough-kms.rs
+++ b/src/bin/roughenough-kms.rs
@@ -26,7 +26,7 @@ extern crate simple_logger;
 extern crate untrusted;
 
 use clap::{App, Arg};
-use roughenough::VERSION;
+use roughenough::roughenough_version;
 
 #[cfg(feature = "awskms")]
 fn aws_kms(kms_key: &str, plaintext_seed: &[u8]) {
@@ -69,7 +69,7 @@ pub fn main() {
     simple_logger::init_with_level(Level::Info).unwrap();
 
     let matches = App::new("roughenough-kms")
-        .version(VERSION)
+        .version(roughenough_version().as_ref())
         .long_about("Encrypt a Roughenough server's long-term seed using a KMS")
         .arg(
             Arg::with_name("KEY_ID")
diff --git a/src/bin/roughenough-server.rs b/src/bin/roughenough-server.rs
index 9778998..d541207 100644
--- a/src/bin/roughenough-server.rs
+++ b/src/bin/roughenough-server.rs
@@ -41,7 +41,7 @@ use std::sync::atomic::Ordering;
 use roughenough::config;
 use roughenough::config::ServerConfig;
 use roughenough::server::Server;
-use roughenough::VERSION;
+use roughenough::roughenough_version;
 
 macro_rules! check_ctrlc {
     ($keep_running:expr) => {
@@ -93,26 +93,12 @@ fn polling_loop(config: Box<ServerConfig>) {
     }
 }
 
-fn kms_support_str() -> &'static str {
-    if cfg!(feature = "awskms") {
-        " (+AWS KMS)"
-    } else if cfg!(feature = "gcpkms") {
-        " (+GCP KMS)"
-    } else {
-        ""
-    }
-}
-
 pub fn main() {
     use log::Level;
 
     simple_logger::init_with_level(Level::Info).unwrap();
 
-    info!(
-        "Roughenough server v{}{} starting",
-        VERSION,
-        kms_support_str()
-    );
+    info!("Roughenough server v{} starting", roughenough_version());
 
     let mut args = env::args();
     if args.len() != 2 {
diff --git a/src/lib.rs b/src/lib.rs
index 614dd90..b87f800 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -86,6 +86,19 @@ pub use tag::Tag;
 /// Version of Roughenough
 pub const VERSION: &str = "1.1.0";
 
+/// Roughenough version string enriched with any compile-time optional features
+pub fn roughenough_version() -> String {
+    let kms_str = if cfg!(feature = "awskms") {
+        " (+AWS KMS)"
+    } else if cfg!(feature = "gcpkms") {
+        " (+GCP KMS)"
+    } else {
+        ""
+    };
+
+    format!("{}{}", VERSION, kms_str)
+}
+
 //  Constants and magic numbers of the Roughtime protocol
 
 /// Minimum size (in bytes) of a client request