patch 7.4.1191

Problem:    The channel feature isn't working yet.
Solution:   Add the connect(), disconnect(), sendexpr() and sendraw()
            functions.  Add initial documentation.  Add a demo server.

4 files changed, 345 insertions, 1 deletions

diff --git a/runtime/doc/Makefile b/runtime/doc/Makefile
index dc49bb7e7..6bbcbc367 100644
--- a/runtime/doc/Makefile
+++ b/runtime/doc/Makefile
@@ -17,6 +17,7 @@ DOCS = \
 	arabic.txt \
 	autocmd.txt \
 	change.txt \
+	channel.txt \
 	cmdline.txt \
 	debug.txt \
 	debugger.txt \
@@ -151,6 +152,7 @@ HTMLS = \
 	arabic.html \
 	autocmd.html \
 	change.html \
+	channel.html \
 	cmdline.html \
 	debug.html \
 	debugger.html \
diff --git a/runtime/doc/channel.txt b/runtime/doc/channel.txt
new file mode 100644
index 000000000..21ce4ebda
--- /dev/null
+++ b/runtime/doc/channel.txt
@@ -0,0 +1,218 @@
+*channel.txt*      For Vim version 7.4.  Last change: 2016 Jan 28
+
+
+		  VIM REFERENCE MANUAL	  by Bram Moolenaar
+
+
+		      Inter-process communication		*channel*
+
+DRAFT  DRAFT  DRAFT  DRAFT  DRAFT  DRAFT  DRAFT  DRAFT  DRAFT  DRAFT
+
+Vim uses channels to communicate with other processes.
+A channel uses a socket.				*socket-interface*
+
+Vim current supports up to 10 simultanious channels.
+The Netbeans interface also uses a channel. |netbeans|
+
+1. Demo					|channel-demo|
+2. Opening a channel			|channel-open|
+3. Using a JSON channel			|channel-use|
+4. Vim commands				|channel-commands|
+5. Using a raw channel			|channel-use|
+6. Job control				|job-control|
+
+{Vi does not have any of these features}
+{only available when compiled with the |+channel| feature}
+
+==============================================================================
+1. Demo							*channel-demo*
+
+This requires Python.  The demo program can be found in
+$VIMRUNTIME/tools/demoserver.py
+Run it in one terminal.  We will call this T1.
+
+Run Vim in another terminal.  Connect to the demo server with: >
+	let handle = connect('localhost:8765', 'json')
+
+In T1 you should see:
+	=== socket opened === ~
+
+You can now send a message to the server: >
+	echo sendexpr(handle, 'hello!')
+
+The message is received in T1 and a response is sent back to Vim.
+You can see the raw messages in T1.  What Vim sends is:
+	[1,"hello!"] ~
+And the response is:
+	[1,"got it"] ~
+The number will increase every time you send a message.
+
+The server can send a command to Vim.  Type this on T1 (literally, including
+the quotes): >
+	NOT IMPLEMENTED YET
+	["ex","echo 'hi there'"]
+And you should see the message in Vim.
+
+To handle asynchronous communication a callback needs to be used: >
+	func MyHandler(handle, msg)
+	  echo "from the handler: " . a:msg
+	endfunc
+	call sendexpr(handle, 'hello!', "MyHandler")
+
+Instead of giving a callback with every send call, it can also be specified
+when opening the channel: >
+	call disconnect(handle)
+	let handle = connect('localhost:8765', 'json', "MyHandler")
+	call sendexpr(handle, 'hello!', 0)
+
+==============================================================================
+2. Opening a channel					*channel-open*
+
+To open a channel:
+    let handle = connect({address}, {mode}, {callback})
+
+{address} has the form "hostname:port".  E.g., "localhost:8765".
+
+{mode} can be:						*channel-mode*
+	"json" - Use JSON, see below; most convenient way
+	"raw"  - Use raw messages
+
+							*channel-callback*
+{callback} is a function that is called when a message is received that is not
+handled otherwise.  It gets two arguments: the channel handle and the received
+message. Example: >
+	func Handle(handle, msg)
+	  echo 'Received: ' . a:msg
+	endfunc
+	let handle = connect("localhost:8765", 'json', "Handle")
+
+When {mode} is "json" the "msg" argument is the body of the received message,
+converted to Vim types.
+When {mode} is "raw" the "msg" argument is the whole message as a string.
+
+When {mode} is "json" the {callback} is optional.  When omitted it is only
+possible to receive a message after sending one.
+
+The handler can be added or changed later: >
+    call sethandler(handle, {callback})
+When {callback} is empty (zero or an empty string) the handler is removed.
+
+Once done with the channel, disconnect it like this: >
+    call disconnect(handle)
+
+==============================================================================
+3. Using a JSON channel					*channel-use*
+
+If {mode} is "json" then a message can be sent synchronously like this: >
+    let response = sendexpr(handle, {expr})
+This awaits a response from the other side.
+
+To send a message, without handling a response: >
+    call sendexpr(handle, {expr}, 0)
+
+To send a message and letting the response handled by a specific function,
+asynchronously: >
+    call sendexpr(handle, {expr}, {callback})
+
+The {expr} is converted to JSON and wrapped in an array.  An example of the
+message that the receiver will get when {expr} is the string "hello":
+	[12,"hello"] ~
+
+The format of the JSON sent is:
+    [{number},{expr}]
+
+In which {number} is different every time.  It must be used in the response
+(if any):
+
+    [{number},{response}]
+
+This way Vim knows which sent message matches with which received message and
+can call the right handler.  Also when the messages arrive out of order.
+
+The sender must always send valid JSON to Vim.  Vim can check for the end of
+the message by parsing the JSON.  It will only accept the message if the end
+was received.
+
+When the process wants to send a message to Vim without first receiving a
+message, it must use the number zero:
+    [0,{response}]
+
+Then channel handler will then get {response} converted to Vim types.  If the
+channel does not have a handler the message is dropped.
+
+On read error or disconnect() the string "DETACH" is sent, if still possible.
+The channel will then be inactive.
+
+==============================================================================
+4. Vim commands						*channel-commands*
+
+NOT IMPLEMENTED YET
+
+With a "json" channel the process can send commands to Vim that will be
+handled by Vim internally, it does not require a handler for the channel.
+
+Possible commands are:
+    ["ex",     {Ex command}]
+    ["normal", {Normal mode command}]
+    ["eval",   {number}, {expression}]
+    ["expr",   {expression}]
+
+With all of these: Be careful what these commands do!  You can easily
+interfere with what the user is doing.  To avoid trouble use |mode()| to check
+that the editor is in the expected state.  E.g., to send keys that must be
+inserted as text, not executed as a command: >
+    ["ex","if mode() == 'i' | call feedkeys('ClassName') | endif"]
+
+The "ex" command is executed as any Ex command.  There is no response for
+completion or error.  You could use functions in an |autoload| script.
+You can also invoke |feedkeys()| to insert anything.
+
+The "normal" command is executed like with |:normal|.
+
+The "eval" command will result in sending back the result of the expression:
+	[{number}, {result}]
+Here {number} is the same as what was in the request.
+
+The "expr" command is similar, but does not send back any response.
+Example:
+	["expr","setline('$', ['one', 'two', 'three'])"]
+
+==============================================================================
+5. Using a raw channel					*channel-raw*
+
+If {mode} is "raw" then a message can be send like this: >
+    let response = sendraw(handle, {string})
+The {string} is sent as-is.  The response will be what can be read from the
+channel right away.  Since Vim doesn't know how to recognize the end of the
+message you need to take care of it yourself.
+
+To send a message, without expecting a response: >
+    call sendraw(handle, {string}, 0)
+The process can send back a response, the channel handler will be called with
+it.
+
+To send a message and letting the response handled by a specific function,
+asynchronously: >
+    call sendraw(handle, {string}, {callback})
+
+This {string} can also be JSON, use |jsonencode()| to create it and
+|jsondecode()| to handle a received JSON message.
+
+==============================================================================
+6. Job control						*job-control*
+
+NOT IMPLEMENTED YET
+
+To start another process: >
+    call startjob({command})
+
+This does not wait for {command} to exit.
+
+TODO:
+
+    let handle = startjob({command}, 's')            # uses stdin/stdout
+    let handle = startjob({command}, '', {address})  # uses socket
+    let handle = startjob({command}, 'd', {address}) # start if connect fails
+
+
+ vim:tw=78:ts=8:ft=help:norl:
diff --git a/runtime/doc/eval.txt b/runtime/doc/eval.txt
index 42503197d..959098abb 100644
--- a/runtime/doc/eval.txt
+++ b/runtime/doc/eval.txt
@@ -1820,6 +1820,8 @@ complete_add( {expr})		Number	add completion match
 complete_check()		Number	check for key typed during completion
 confirm( {msg} [, {choices} [, {default} [, {type}]]])
 				Number	number of choice picked by user
+connect( {address}, {mode} [, {callback}])
+				Number	open a channel
 copy( {expr})			any	make a shallow copy of {expr}
 cos( {expr})			Float	cosine of {expr}
 cosh( {expr})			Float	hyperbolic cosine of {expr}
@@ -2027,6 +2029,10 @@ searchpairpos( {start}, {middle}, {end} [, {flags} [, {skip} [...]]])
 				List	search for other end of start/end pair
 searchpos( {pattern} [, {flags} [, {stopline} [, {timeout}]]])
 				List	search for {pattern}
+sendexpr( {handle}, {expr} [, {callback}])
+				any	send {expr} over JSON channel {handle}
+sendraw( {handle}, {string} [, {callback}])
+				any	send {string} over raw channel {handle}
 server2client( {clientid}, {string})
 				Number	send reply string
 serverlist()			String	get a list of available servers
@@ -2660,6 +2666,18 @@ confirm({msg} [, {choices} [, {default} [, {type}]]])
 		don't fit, a vertical layout is used anyway.  For some systems
 		the horizontal layout is always used.
 
+connect({address}, {mode} [, {callback}])		*connect()*
+		Open a channel to {address}.  See |channel|.
+
+		{address} has the form "hostname:port", e.g.,
+		"localhost:8765".
+
+		{mode} is either "json" or "raw".  See |channel-mode| for the
+		meaning.
+
+		{callback} is a function that handles received messages on the
+		channel.  See |channel-callback|.
+
 							*copy()*
 copy({expr})	Make a copy of {expr}.	For Numbers and Strings this isn't
 		different from using {expr} directly.
@@ -3861,7 +3879,9 @@ glob2regpat({expr})					 *glob2regpat()*
 			if filename =~ glob2regpat('Make*.mak')
 <		This is equivalent to: >
 			if filename =~ '^Make.*\.mak$'
-<
+<		When {expr} is an empty string the result is "^$", match an
+		empty string.
+
 								*globpath()*
 globpath({path}, {expr} [, {nosuf} [, {list} [, {allinks}]]])
 		Perform glob() on all directories in {path} and concatenate
@@ -5593,6 +5613,23 @@ searchpos({pattern} [, {flags} [, {stopline} [, {timeout}]]])	*searchpos()*
 <		In this example "submatch" is 2 when a lowercase letter is
 		found |/\l|, 3 when an uppercase letter is found |/\u|.
 
+sendexpr({handle}, {expr} [, {callback}])		*sendexpr()*
+		Send {expr} over JSON channel {handle}.  See |channel-use|.
+
+		When {callback} is given returns immediately.  Without
+		{callback} waits for a JSON response and returns the decoded
+		expression.  When there is an error or timeout returns an
+		empty string.
+
+		When {callback} is zero no response is expected.
+		Otherwise {callback} must be a Funcref or the name of a
+		function.  It is called when the response is received.  See
+		|channel-callback|.
+
+sendraw({handle}, {string} [, {callback}])		*sendraw()*
+		Send {string} over raw channel {handle}.  See |channel-raw|.
+		Works like |sendexpr()|, but does not decode the response.
+
 server2client( {clientid}, {string})			*server2client()*
 		Send a reply string to {clientid}.  The most recent {clientid}
 		that sent a string can be retrieved with expand("<client>").
diff --git a/runtime/tools/demoserver.py b/runtime/tools/demoserver.py
new file mode 100644
index 000000000..c72a58b73
--- /dev/null
+++ b/runtime/tools/demoserver.py
@@ -0,0 +1,87 @@
+#!/usr/bin/python
+# Server that will accept connections from a Vim channel.
+# Run this server and then in Vim you can open the channel:
+#  :let handle = connect('localhost:8765', 'json')
+#
+# Then Vim can send requests to the server:
+#  :let response = sendexpr(handle, 'hello!')
+#
+# And you can control Vim by typing a JSON message here, e.g.:
+#   ["ex","echo 'hi there'"]
+#
+# See ":help channel-demo" in Vim.
+
+import SocketServer
+import json
+import socket
+import sys
+import threading
+
+thesocket = None
+
+class ThreadedTCPRequestHandler(SocketServer.BaseRequestHandler):
+
+    def handle(self):
+        print "=== socket opened ==="
+        global thesocket
+        thesocket = self.request
+        while True:
+            try:
+                data = self.request.recv(4096)
+            except socket.error:
+                print "=== socket error ==="
+                break
+            except IOError:
+                print "=== socket closed ==="
+                break
+            if data == '':
+                print "=== socket closed ==="
+                break
+            print "received: {}".format(data)
+            try:
+                decoded = json.loads(data)
+            except ValueError:
+                print "json decoding failed"
+                decoded = [0, '']
+
+            if decoded[1] == 'hello!':
+                response = "got it"
+            else:
+                response = "what?"
+            encoded = json.dumps([decoded[0], response])
+            print "sending {}".format(encoded)
+            self.request.sendall(encoded)
+        thesocket = None
+
+class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
+    pass
+
+if __name__ == "__main__":
+    HOST, PORT = "localhost", 8765
+
+    server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
+    ip, port = server.server_address
+
+    # Start a thread with the server -- that thread will then start one
+    # more thread for each request
+    server_thread = threading.Thread(target=server.serve_forever)
+
+    # Exit the server thread when the main thread terminates
+    server_thread.daemon = True
+    server_thread.start()
+    print "Server loop running in thread: ", server_thread.name
+
+    print "Listening on port {}".format(PORT)
+    while True:
+        typed = sys.stdin.readline()
+        if "quit" in typed:
+            print "Goodbye!"
+            break
+        if thesocket is None:
+            print "No socket yet"
+        else:
+            print "sending {}".format(typed)
+            thesocket.sendall(typed)
+
+    server.shutdown()
+    server.server_close()