<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Feb 18, 2014 at 7:54 AM, ianG <span dir="ltr"><<a href="mailto:iang@iang.org" target="_blank">iang@iang.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I think we can do a lot lot better. I have a document somewhere on<br>
this, but in brief:<br>
<br>
There are too many primitives. I see 11 doing numbers alone! In<br>
practice, in network protocols, we do not need bignums, we do not need<br>
floats and we do not need negatives. </blockquote><div><br></div><div>But part of the exercise here is to try to get to convergence on one single encoding by meeting all needs. I don't see a lot of need for compression or for 128 bit floats. But some people need them for their data work. Hence JSON-C and JSON-D.</div>
<div><br></div><div>I understand the temptation to go optimizing bits, but I don't think there is any point in my current applications which are JSON Web Services over HTTP. However, right at the end of this over-long message I show a situation where bit grinding does provide real value.</div>
<div><br></div><div><br></div><div>What we could do is to either define a profile that drops unneeded features or have applications state which features they actually use.</div><div><br></div><div>If we were doing a crypto framing scheme then it would make sense to say that the bits on the wire must only use the binary tags and that only positive integers, strings and binary data chunks are used. But the documentation could still use the JSON representation for illustrative purposes/</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Then, for different sized numbers,<br>
we should remember that we are about simplification and higher level<br>
concepts. Which should tell us we need a number. Not the four<br>
remaining of 8, 16, 32, 64 bits which are hangovers from the hardware<br>
days. We need one number that simply expands to fill the needs. This<br>
is done with the 7 bit trick, where the high bit being set says there is<br>
a following extra byte.<br></blockquote><div><br></div><div>The problem with the 7 bit trick is that it requires the following bytes to be shifted about, same with the similar UTF8 scheme. The JSON data model is simple enough not to need more than one byte for the tag and length of data to follow combined.</div>
<div><br></div><div>Allowing any number of bytes in an integer from 1 to 8 requires 3 bits which is a lot. Restricting the number of bytes to 1,2,4,8 saves a bit. But that is all that is going on. Using a 16 bit representation on the wire does not mean that the corresponding data model representation is 16 bit. It is just that the number either does or does not fit into 16 bits.</div>
<div><br></div><div>Since the JSON data model has negative integers it is necessary to deal with signs. Which means either twos compliment or a sign bit. I prefer a sign bit because it is a lot simpler to code and avoids confusion between representations.</div>
<div><br></div><div>So I would argue that there are actually only two number representations in JSON-B, Integer and float. </div><div><br></div><div><br></div><div>If a protocol involves integers bigger than 32 bits then ultra-compact representations are probably not a priority. That is why a 16 bit length is used for bigints, while it is virtually certain that a bigint will fit into a 8 bit length, saving the extra byte is not worth the extra code point. </div>
<div><br></div><div>I agree that floating point values are not necessary for protocols and arguably neither are negative integers. But they are part of the JSON data model which means they have to be supported.</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Next, we should really be thinking in OO terms. When we are dealing in<br>
OO, we have a single object that 'knows' its output, and its input<br>
intimately. Which is to say, it knows whereas your spec does not. The<br>
object can do semantics such as range checking and small composition<br>
such as conversion of byte arrays to strings.<br></blockquote><div><br></div><div>Well the code is in C# so that is how the encoder/decoder works. Not sure what you are getting at here.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
We do however need a sequence of bytes, and a byte array is constructed<br>
simply with a length (number as above) and a sequence of bytes. That's<br>
2 primitives so far.<br></blockquote><div><br></div><div>Sounds like you are doing a compact binary version of LISP S-Expressions. Which is another totally valid approach.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
One more thing. Without a seriously good testing and debugging system,<br>
this whole area breaks down with complexity, which is why people eschew<br>
binary and go for text. There is a loopback technique I use to solve<br>
this issue, which I call the Ouroboros pattern. In short it is this:<br>
<br>
1. each object has an example() method which produces a correct object<br>
with with each field constructed randomly.<br>
2. write that out through the stream process.<br>
3. read it back in to a new object, using the converse stream process.<br>
4. compare the two objects with standard equals() method.<br>
<br>
Run 2^5 times for each class. Due to composition and repeated tests<br>
this solves the complexity issue.<br></blockquote><div><br></div><div>Yep, regression testing. Have not got round to writing this yet but I plan to. I wanted to re-implement the scheme in C first. Then I can generate the same data inputs for C and C# and check that they produce the same outputs and decoding produces the original data.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Probably the key point here is that if you are sill thinking about<br>
protocols along the old bits & Bytes way that Richard highlighted,<br>
you're missing out. Doing protocols with OO thinking is so much easier<br>
you never ever go back. Once you do that, all of the formats, ideas,<br>
layouts, MLs start to look a little .. 20th century, steampunk, historical.<br></blockquote><div><br></div><div>I am not sure about the OO thing since I was doing OO back in the days when it really was message passing and then C++ came along and made a mess of the ideas.<br>
</div><div><br></div><div>What we are really doing with a protocol compiler is writing the on the wire format for messages passed between concurrent network objects.</div><div><br></div><div>Further, in a message framework, those messages correspond to method invocations on the objects. So the (abreviated) signature of a Transaction in the Confirmation protocol is:</div>
</div><div><br></div><div><div> Transaction Enquirer Confirmation ConfirmationRequest ConfirmationResponse</div><div> Description</div><div> |Post a request for confirmation to a user.</div><div> Status Success</div>
<div> Status Refused</div><div> Status UnknownUser</div><div><br></div><div> Message ConfirmationRequest</div><div> Description</div><div> |Request a confirmation from a specified user.</div>
<div> String Account</div><div> Required</div><div> Description</div><div> |The user being asked to provide confirmation.</div><div> Description</div><div> |The format of the account identifier is the same as for email,</div>
<div> |i.e. <username>@<domain></div><div> String Text</div><div> Required</div><div> String Option</div><div> Multiple</div></div><div><br></div>
<div>When this is translated into C# we get the following class (abreviated)</div><div><br></div><div><div><span class="" style="white-space:pre"> </span>public partial class ConfirmationRequest : CNF {</div><div><span class="" style="white-space:pre"> </span>public string<span class="" style="white-space:pre"> </span>Account;</div>
<div><span class="" style="white-space:pre"> </span>public string<span class="" style="white-space:pre"> </span>Text;</div><div><span class="" style="white-space:pre"> </span>public List<string><span class="" style="white-space:pre"> </span>Option;</div>
</div><div><br></div><div>Or in C, the following structure:</div><div><br></div><div><div>typedef struct _CNF_ConfirmationRequest {</div><div><span class="" style="white-space:pre"> </span>struct _CNF_ConfirmationRequest<span class="" style="white-space:pre"> </span>*_Next;</div>
<div><span class="" style="white-space:pre"> </span>int<span class="" style="white-space:pre"> </span>_Type;</div><div><span class="" style="white-space:pre"> </span>JSON_String<span class="" style="white-space:pre"> </span>Account;</div>
<div><span class="" style="white-space:pre"> </span>JSON_String<span class="" style="white-space:pre"> </span>Text;</div><div><span class="" style="white-space:pre"> </span>JSON_Group<span class="" style="white-space:pre"> </span>Option;</div>
<div><span class="" style="white-space:pre"> </span>} CNF_ConfirmationRequest;</div></div><div><br></div><div>It turns out that most IETF protocols can be implemented as a sequence of RPC calls between a client and a server. The main exceptions being Jabber (XMPP) and IRC which are essentially reverse RPC calls. The client opens up a context and then receives a series of RPC calls from the server. I have looked into that but to support that communication pattern in a web service I either need to use an extended HTTP that supports multiple streams (e.g. HTTP/2.0) or write my own.</div>
<div><br></div><div><br></div><div>Although PROTOGEN was originally designed to be a protocol synthesizer just for JSON, writing a C library means that I need to be able to make HTTP calls out which in turn means either calling someone else's API and deal with a different implementation of strings etc. etc. and a lot of code I don't need or writing my own very simple HTTP implementation.</div>
<div><br></div><div>So last night I wrote a PROTOGEN schema for HTTP (or at least the parts I care about) and I hope to have it done this week.</div><div><br></div><div><div>Protocol Goedel.HTTP HTTPG</div><div><br></div>
<div><span class="" style="white-space:pre"> </span>Structure Common</div><div><span class="" style="white-space:pre"> </span>String<span class="" style="white-space:pre"> </span>Content_Type</div><div><span class="" style="white-space:pre"> </span>Integer<span class="" style="white-space:pre"> </span>Content_Length</div>
<div><span class="" style="white-space:pre"> </span>DateTime<span class="" style="white-space:pre"> </span>Date</div><div><br></div><div><span class="" style="white-space:pre"> </span>Structure Request</div><div><span class="" style="white-space:pre"> </span>Inherits<span class="" style="white-space:pre"> </span>Common</div>
<div><span class="" style="white-space:pre"> </span>String<span class="" style="white-space:pre"> </span>Accept</div><div><span class="" style="white-space:pre"> </span>Multiple</div><div><span class="" style="white-space:pre"> </span>String<span class="" style="white-space:pre"> </span>Authorization</div>
<div><span class="" style="white-space:pre"> </span>Multiple</div></div><div>[etc]</div><div><br></div><div>One side product of this work is that I can now encode HTTP messages in JSON syntax rather than the traditional RFC822 style headers. Which in turn means that I can write a Web service that only uses one encoding for both the wrapper and the contents and since the compiler supports JSON-B (or soon will) I can optimize a protocol for space by simply setting the encoding flag to JSON-B rather than JSON.</div>
<div><br></div><div>If we went a step further and encoded the SSL layer in the same encoding, ditching the ad-hoc SSL/TLS encodings this would provide consistency all the way down to the IP layer.</div><div><br></div><div>
At this point Ian's proposal for a totally minimal binary encoding with no frills suddenly looks very attractive. The HTTP layers don't need floats or the like so ditch them.</div><div><br></div>-- </div><div class="gmail_extra">
Website: <a href="http://hallambaker.com/" target="_blank">http://hallambaker.com/</a><br>
</div></div>