I’d planned this blog entry to be about the changes I’m making to the myc4d customer portal but over the past few days it has become clear that there is quite a lot of interest in our new kbmMW transports that are in development.  To save making the same comments over and over on the newsgroups I’ve decided to blog about what we’re up to and take the wraps off them instead.

 

There will be a lot of detail covered but some facts will remain un-answered for the simple reason that some things I do not know the answer to yet J

 

I’m going to do this in a semi-Q&A style to get my headings out.  Have fun!

What is the new kbmMW transport?

 

Let’s kill this one off straight away.  The new transport is not a single new transport.  It is a set of transports built on top of our own socket layer removing the need to use third party communication stacks such as Indy or Synapse etc.

 

What is the motivation to build our own communications stack?

The motivation behind the project was driven by a variety of sources. 

 

• There is growing discontent in our newsgroups about using the Indy components as the underlying communications mechanism.  Reasons given vary but within C4D we get frustrated with breaking changes made to the source code over which we have no control.  This is an issue when integrating with any third party but it seems particularly true of Indy between simple point releases.  kbmMW is only using a relatively small set of Indy APIs but we get repeatedly asked how to get the framework running with Indy version X, or Y, or the latest development snapshot etc.  We have transports based on Synapse and have never needed to update these.
• There were requests being made for us to supply messaging transports based on Synapse, motivated partly by the above issues with Indy.
• The current transports have scalability limits.

 

Are you saying kbmMW is not scalable?

I should expand on the last point.  kbmMW is a hugely scalable framework.  I am not implying anything to the contrary with the last statement.  A single server will scale up to many hundreds of users without problem.  This is great for most applications that use the traditional request response operational model – your typical CMS application etc. 

 

When we introduced messaging the framework took on a paradigm shift.  To people building standard applications this will go largely unnoticed but internally the framework behaviour has changed to work in terms of discrete messages.  When applications are built using the messaging transports developers can still call services just like they did before.  But the way these calls are expressed is different.  A request is simply a message sent from a client with the first part of its subject being REQ.  REQ stands for request funnily enough.  The framework takes this message and decodes it into a call to a certain service/method pair and processes the request.  Results from the request are sent back to the caller using another message with the subject RES.  RES stands for response.  I’m digressing a little but the point is people can use the messaging transports without losing any existing functionality.  In fact it opens up huge possibilities when we add intelligent routing to these REQ messages.  Messaging is incredibly powerful!

 

What else can people gain from using messaging then?  

To use a cliché, access to a whole new world of applications where discrete message passing is the key.  I have been involved in a project requiring the distribution of real-time information to huge numbers of clients or potential clients.  Majority of the time the clients may be doing nothing other than monitoring some statistical feeds.  The amount of data they receive is small but there are a lot of them.  By this I’m talking about potentially millions.  The question is how to support millions of client connections to the kbmMW messaging based WIB?

 

It is time for another digression – why can’t we use the Indy messaging transport?  To answer this question I need to explain how the Indy, Synapse and DXSocks TCPIP based transports work.  The focus of this is the server or hub in messaging terms.

 

Each client connects to the hub using a TCPIP socket.  Data is sent from the client and transferred by the TCPIP stack over to the server where it is presented as a memory buffer to be read.  But how do we know there is data ready for us to read?  Simple – we call various APIs to tell us if there is a buffer waiting for us.  That is the problem.  Read it again, “We call various APIs to tell us if there is a buffer waiting for us”.  We have to poll the status of the socket.  To do this the usual arrangement is to have a thread executing in a loop polling the socket.

 

Here is the code pattern for the server transports.

 

While the client is connected

            Test socket for data

            If data is present

                        Process data

            Else

                        Sleep for a moment 

In the Indy transport their source implements the loop.  For Synapse, which is API based, we implement it in ourselves.

 

A quick quiz!

 

Q. How many threads does it take to service 100 clients?

A. 101  (There is also a separate thread responsible for listening for new connections and creating the new socket/thread pair). 

 

Q. How many threads does it take to service 500 clients?

A. 501

 

Q. How many threads does it take to service 10000 clients?

A. Too many!

 

As more and more threads are created there is more and more contention for CPU resource.  This takes the form of expensive thread context switches and there comes a point where the majority of CPU time is spent simply switching from thread to thread.  The official description is thread thrashing. Throughput of the server decays rapidly.

 

How do we avoid thread thrashing?

The answer is relatively obvious – use less threads!  One approach that can be used is to have each thread handling more than one socket but this is still essentially a polling solution.  We need something else.

 

Welcome to IO Completion Ports

This was introduced in Windows NT 3.5 and is a complete paradigm shift.  Instead of polling sockets for data we instead get the TCPIP stack to notify us when new data has arrived.  It sounds obvious and simple.  Of course it doesn’t work out to be quite that simple. The goal in any server design should be to try and incur as few context switches as possible by avoiding threads being unnecessarily blocked whilst at the same time maximising parallelism using multiple threads.  The ideal situation is to have a thread actively servicing a client request on every processor and for those threads not to block in the event that additional requests are waiting when they complete their current request.  For this to work there must be a way for the application to activate another thread when others are busy processing another I/O operation.  Windows NT 3.5 introduced a set of APIs based on something called the completion port.  Applications can associate a completion port with a TCPIP socket.  Data coming in off the socket results in a completion packet being queued to the port for processing.  A set of threads associated with the port read off the data packet and process it.  When a completion port is created we specify a concurrency value.  This is the maximum number of threads that can be actively processing data packets queued to the port at any one time.  The Windows Kernel performs the creation and management of these threads.  The aim is to have one thread active at any given time per processor.  A typical rule of thumb for the concurrency value is 2 times the number of CPU cores.  At this time we have an open mind on this until we perform some more scalability tests.  The Windows scheduler attempts to reduce context switches by selecting the same thread to process request N + 1 after processing request N this allows CPUs to be utilised to near their full capacity.

 

 

Running the quick quiz again!

 

For a 4 CPU server 

Q. How many IO worker threads does it take to service 100 clients?

A. 8 using rule of thumb  

Q. How many threads does it take to service 500 clients?

A. 8 using rule of thumb  

Q. How many threads does it take to service 10000 clients?

A. 8 using rule of thumb  

What a result!

It’s not just about threads.

We have a mechanism to reduce the thread count and avoid thread thrashing by using our completion port.  But there is another effect we can overcome by having our own socket layer.

 

As competent and complete as they are Indy and Synapse are complete communication stacks.  By this I mean they present a nice uniform view of the data coming off the socket.  When we detect in our listener thread that there is data on the socket we are presented with the entire stream of data sent over from the client.  The buffer size can range from a few bytes to multi-megabytes. kbmMW must then take a copy of that buffer in order to safely work with it.  That is a copy operation.

 

IOCP works slightly differently.  It is not just the IO worker threads that are associated with the completion port; the socket is also using a special data structure of type WSAOverlapped.  We can see from the diagram above that somehow data passes through the completion port from the TCPIP stack and gets processed by the IO worker thread.  The data passes through by being copied to memory buffers that we supply.  We supply the completion port with our memory buffers by posting them to the completion port using the WSAOverlapped structure associated with the socket.  What this means is that the IO worker threads can copy data into our buffers straight off the TCPIP stack without needing a context switch.  If we supply enough buffers to the completion port before the client request comes in, then the whole request can be copied to buffers without a context switch.  In addition if we’re processing a request in the framework and another request (read message actually) comes in from the client on that socket another IO thread can start processing it in parallel. 

 

What does this mean?  It means that we are removing the large buffer copy of the complete stream that we have to perform for Indy/Synapse.  This will boost performance.  In addition if we manage the number of buffer we post to the completion port well we can allow the IO worker thread to pull data off the TCPIP stack unhindered.  It is worth remembering that the IO worker threads are now running in kernel mode not user mode.   In essence – kbmMW is moving closer to the TCPIP stack with the removal of Indy/Synapse.

 

I’ve been writing for a while now and need to get back to some coding.  There will be some further blog entries on IOCP because there is a lot more to talk about.

Keep well

Richard

 

 

 

The demos are built.  The help file is writen.  The installer is done. 

Introducing kbmMW Spider 2 for dotnet

Firstly a big thanks for the field testers.  They made the product better and proven in a live environment including our own myc4d.com and turbomiddleware.com sites.

I thought I'd run through the shipping demos,  they're kind of fun.

There is a W32 appilication server based on the standard BDE demo we ship with the framework.  I decided to implement the classic biolife demo. 

Here is a code sample form it's main service

function TTestQuery.ProcessRequest(const Func: string;
  const ClientIdent: TkbmMWClientIdentity;
  const Args: array of Variant): Variant;
begin
  if (Func='FETCHBIOLIFE') then
    StreamBiolife else
  if (Func='FETCHBIOLIFEOBJECTS') then
    StreamBiolifeObjects
  else
    inherited ProcessRequest(Func,ClientIdent,Args);

end;

procedure TTestQuery.StreamBiolife;
begin
  self.BIOLIFE.Open;
  try
    self.BIOLIFE.SaveToStreamViaFormat(self.ResultStream,self.kbmMWBinaryStreamFormat1);
  finally
    self.BIOLIFE.Close;
  end;
end;

There are two service methods supported.  The first one simply opens BioLife.db and streams the resultset down the wire.

I have three demo clients.  They are

• Winforms client built with VS2005
• VCL.net client built with D2007 for dotnet
• ASP.net client built with VS2005

 

Here is the important code for the Winforms sample

        private void button1_Click(object sender, EventArgs e)
        {
            C4D.kbmMW.Transports.TCPIPTransport transport = new C4D.kbmMW.Transports.TCPIPTransport();
            transport.Port = 3000;
            transport.Host = "127.0.0.1";

            transport.Connect();

            C4D.kbmMW.Client.SimpleClient client = new C4D.kbmMW.Client.SimpleClient(transport);

            C4D.kbmMW.Data.DataAdapter da = new C4D.kbmMW.Data.DataAdapter();
            da.FormatterType = C4D.Common.Formatters.FormatterType.kbmMW;

            client.SendRequest("KBMMW_QUERY", "", "FETCHBIOLIFE");

            System.Data.DataTable dataTable = new DataTable("biolife");

            da.FillDatatable(dataTable, client.ResultStream);

            dataGrid.DataSource = dataTable;
            dataGrid.Columns["Graphic"].AutoSizeMode = DataGridViewAutoSizeColumnMode.AllCells;
        }

And here is the output

 

Cool

Here is the VCL.net code sample

procedure TForm1.Button1Click(Sender: TObject);
var
  transport: C4D.kbmMW.Transports.TCPIPTransport;
  client: C4D.kbmMW.Client.SimpleClient;
  da: C4D.kbmMW.Data.DataAdapter;
  dataTable: System.Data.DataTable;
  ADOBridge: Borland.Vcl.ADONETDb.TADONETConnector;
begin
  transport:= C4D.kbmMW.Transports.TCPIPTransport.Create;
  transport.Port:= 3000;
  transport.Host:= '127.0.0.1';

  transport.Connect();

  client:= C4D.kbmMW.Client.SimpleClient.Create(transport);

  da:= C4D.kbmMW.Data.DataAdapter.Create;
  da.FormatterType:= C4D.Common.Formatters.FormatterType.kbmMW;

  client.SendRequest('KBMMW_QUERY', '', 'FETCHBIOLIFE');

  dataTable:= System.Data.DataTable.Create('biolife');

  da.FillDatatable(dataTable, client.ResultStream);

  ADOBridge:= Borland.Vcl.ADONETDb.TADONETConnector.Create(nil);
  ADOBridge.DataTable:= dataTable;

  self.DataSource.DataSet:= ADOBridge;
  self.DBGrid1.DataSource:= self.DataSource;

end;

and the output 

 

 

The Delphi grid doesn't recognise the images - but owner draw would cope with that.

Here's the ASP.net VS2005 sample.

    protected void Page_Load(object sender, EventArgs e)
    {
        //  We're using a connection pool for this sample, see Global.cs
        C4D.kbmMW.Client.PooledSimpleClient client = new C4D.kbmMW.Client.PooledSimpleClient(Global.ConnectionPool);

        C4D.kbmMW.Data.DataAdapter da = new C4D.kbmMW.Data.DataAdapter();
        da.FormatterType = C4D.Common.Formatters.FormatterType.kbmMW;

        client.SendRequest("KBMMW_QUERY", "", "FETCHBIOLIFE");

        System.Data.DataTable dataTable = new DataTable("biolife");

        da.FillDatatable(dataTable, client.ResultStream);

        this.GridView1.DataSource = dataTable;
        this.GridView1.DataBind();

    }

You will notice the use of Global.ConnectionPool

This is the connection pooling feature of Spider2 and is setup like this in Global.cs

public partial class Global : System.Web.HttpApplication
{
    private static C4D.kbmMW.Client.ClientConnectionPool connectionPool;
    public Global()
    {
        connectionPool = new C4D.kbmMW.Client.ClientConnectionPool();
        C4D.kbmMW.Transports.TCPIPTransport transport = new C4D.kbmMW.Transports.TCPIPTransport();
        transport.Host = "127.0.0.1";
        transport.Port = 3000;
        connectionPool.Template = transport;
    }

    public static C4D.kbmMW.Client.ClientConnectionPool ConnectionPool
    {
        get { return connectionPool; }
    }

    protected void Application_Start(Object sender, EventArgs e)
    {
        connectionPool.Active = true;
    }
 }

 

 

and the output

Finally, the DataAdapters use the VCL streaming support.  There is nothing to stop us using this directly.

The second method on the server was

procedure TTestQuery.StreamBiolifeObjects;
var
  writer: TWriter;
begin
  writer:= TWriter.Create(self.ResultStream,1024);
  self.BIOLIFE.Open;
  try
    writer.WriteInteger(self.BIOLIFE.RecordCount);
    while not self.BIOLIFE.Eof do
    begin
      writer.WriteInteger(self.BIOLIFE.FieldByName('Species No').AsInteger);
      writer.WriteString(self.BIOLIFE.FieldByName('Category').AsString);
      writer.WriteString(self.BIOLIFE.FieldByName('Common_Name').AsString);
      writer.WriteString(self.BIOLIFE.FieldByName('Species Name').AsString);
      writer.WriteFloat(self.BIOLIFE.FieldByName('Length (cm)').AsFloat);
      writer.WriteFloat(self.BIOLIFE.FieldByName('Length_In').AsFloat);
      writer.WriteString(self.BIOLIFE.FieldByName('Notes').AsString);
      self.BIOLIFE.Next;
    end;    // while
  finally
    writer.Free;
    self.BIOLIFE.Close;
  end;
end;

Here we use a TWriter component to write out the Biolife data directly onto the resultstream.

On the ASP.net client we have this code to decode it.

    class BiolifeObject
    {
        private int speciesNo;
        private string category;
        private string commonName;
        private string speciesName;
        private double lengthCM;
        private double lengthIn;
        private string notes;

        public int SpeciesNo
        {
            get { return this.speciesNo; }
            set { this.speciesNo = value; }
        }
        public string Category
        {
            get { return category; }
            set { category = value; }
        }
        public string CommonName
        {
            get { return commonName; }
            set { commonName = value; }
        }
        public string SpeciesName
        {
            get { return speciesName; }
            set { speciesName = value; }
        }
        public double LengthCM
        {
            get { return lengthCM; }
            set { lengthCM = value; }
        }
        public double LengthIn
        {
            get { return lengthIn; }
            set { lengthIn = value; }
        }
        public string Notes
        {
            get { return notes; }
            set { notes = value; }
        }
    }

This is a class to accept the data and we bind it like this.

    protected void Button1_Click(object sender, EventArgs e)
    {
        //  We're using a connection pool for this sample, see Global.cs
        C4D.kbmMW.Client.PooledSimpleClient client = new C4D.kbmMW.Client.PooledSimpleClient(Global.ConnectionPool);
        client.SendRequest("KBMMW_QUERY", "", "FETCHBIOLIFEOBJECTS");

        C4D.Common.VCL.VCLReader reader = new C4D.Common.VCL.VCLReader(client.ResultStream);
        System.Collections.ArrayList objects = new System.Collections.ArrayList();

        int ObjectCount = reader.ReadInteger();

        while (ObjectCount > 0)
        {
            BiolifeObject bo = new BiolifeObject();
            bo.SpeciesNo = reader.ReadInteger();
            bo.Category = reader.ReadString();
            bo.CommonName = reader.ReadString();
            bo.SpeciesName = reader.ReadString();
            bo.LengthCM = reader.ReadFloat();
            bo.LengthIn = reader.ReadFloat();
            bo.Notes = reader.ReadString();
            objects.Add(bo);
            ObjectCount--;
        }

        this.GridView1.DataSource = objects;
        this.GridView1.DataBind();

    }

We marshall the streamed data off the resultstream using a VCLReader and create an arraylist of Biolife objects.  These can be bound to directly from the grid. 

The output is as follows (hit FetchAsObjects), although the property ordering isn't maintained.

There you go and blindingly quick.

Look out for the release announcement.  It will be available at MyC4D.com

Richard

Components4Developers have been given special permission by CodeGear to talk about the Highlander field test release.

In this third blog entry about the CodeGear Highlander pre-release I'm going to demonstrate a full kbmMW native Delphi client in operation.

It is going to call a windows 32bit kbmMW server that has a service supporting three operations for this demo.

1. A simple echo remote procedure call

2. Streaming a TkbmMemtable down the wire to the client

3. Execution of a named query that returns a list of C4D products form a SQL Server 2005 database

Below is the user control I designed in Highlander for the three demos

1. A simple echo remote procedure call

Here is the code for the Ping button

To make things simpler I encapsulated the wiring up of the TCP/IP transport into a connector class to declutter these samples.  This simple demo sends the content of the text box and sends it to our service named KBMMW_QUERY for processing my the method PingServer.  PingServer simply returns the text sent to it as argument 0.  We take the return value and update a lable to the right of out ping button.

Here's the result

 

2. Streaming a TkbmMemtable down the wire to the client

The server being used for these demos is the same one used when developing Spider 2 - our extended C# client.  On our service we have a method called GemMemtable that creates and defines a TkbmMemtable in the fly and populates it with two records.  The whole memtable structure and data is then formatted onto the resultstream and sent down the wire.  At the client we must do the reverse.

For this sample we perform the decode into a local TkbmMemtable and then bind that to a grid to visualise the result.  Here is the code

 

Once again we call our service but this time we execute the method GetMemtable. After the call the data from our server resides in the stream client.resultstream.  We create a formatter, in this case the kbmMW formatter, and populate a TkbmMemtable using it.  This can be seen in the line memtable.LoadFromStreamViaFormat(client.ResultStream,formatter)

After this point we have our TkbmMemtable correctly structured and populated with data from the server.  Because of the dotnet extensions I talked about in blog 2 I can bind this directly to the standard ASP.net Gridview control.

Here is the result:

3. Execution of a named query that returns a list of C4D products form a SQL Server 2005 database

kbmMW users will be familiar with the concept of named queries.  We want to promote good n-tier practices where business knowledge is encapsulated at the server and not the client.  Named queries promote this by allowing clients to execute a query by name without needing any knowledge of the SQL that will execute.

Here is the code:

In this sample we create a TkbmMWClientQuery to accept the data from our server.  We bind it to a formatter, in this case the TkbmMWBinaryStreamFormat, and set it's Query text to be our named query @highlanderDemo.  Next we simply call Open and the rest is automatic.  Once more we bind the result to a gridview.  The named query that our client had no knowledge about lists the products available on myc4d.com.  Here is the result:

One final note to end with and this is mainly of interest to kbmMW users.  During the very early phase of the Highlander FT there was no Indy 10 assembly to use.  In order to progress we wrote our own native Delphi socket client.  Here is a snippet of the client connector encapsulation used for these demos where you will notice the new transport TkbmMWTCPIPDotNetClientTransport :

A final final note.  Highlander has been great in terms of code compatibility.  Almost all the codebase has ported without much effort. Most changes have been due to alterations in the dotnet framework itself.  A good sign.

Richard

 

Components4Developers have been given special permission by CodeGear to talk about the Highlander field test release.

A project that I have running which has slipped on our release schedule is kbmMemtable dotnet extensions.  This uses the standard TkbmMemtable with some special plug-in extensions.  The extensions are relevent to dotnet only and extend the memtable to be fully bindable in dotnet.  I had this running under dotnet 1.1 built using BDS2006 but wanted to take it further by re-optimsing some of the memtable internal structures to boost it's speed and capabilities.  Unfortunately at the time the internal restructuring I had in mind used features only available in dotnet 2.0 hence the slip. 

Now that Highlander has come along I've re-visited my project. After rebuilding my assembly with the plugin extensions I created a new ASP.net project and added a reference to it - kbmMemProD2007dnRun.dll in this case.

 

I then added some code into the page_load event of the default WebForm1.aspx where I create a kbmMemtable and add three fielddefs to it.  These are of type String, Integer and Float.

I then Open the table and add 20 records to it.  My page contains a DataGrid and once I've added my records to the memtable I bind it to the datagrid.

 

Here is a screenshot of the running application:

So - some might wonder what the big deal is?

Well there are a few things going on. 

1) kbmMemtable is running under dotnet 2

2) we're able to bind kbmMemtable as a datasource to standard dotnet controls

3) kbmMemtable is still a TDataset descendent and hence can be bound to any VCL.net control too

4) kbmMW queries are descended from TkbmCustomMemtable - hence kbmMW resultsets also support 1, 2 and 3 above.

In combination with Spider 2 that I blogged about last time I hope you can see it is now possible to pass around and work with kbmMemtables on dotnet 1.1, 2.0 and 3.0, win32 and linux.  More to the point - you can choose to visualise the data using any combination of VCL, VCL.net, WinForms or ASP.net controls/components. 

My next blog will show kbmMW working in Highlander. 

Richard

First the disclaimer

Components4Developers have been given specific permission by CodeGear to discuss Highlander in public.

All screenshots are from a pre-release beta version of Highlander.

I decided to run up the kbmMWSpider extensions for dotnet in Highlander and check that it all worked nicely.  It did and it was great to see Spider and Delphi code back together for DotNet2.

Here is a sequence of screenshots to show you what I did but basically I ran up an ASP.net application, added a page and put a grid and a button on it.  In the OnClick event of the button I make a call out to a kbmMW server and bind the returned data to the grid.

Here we go

1) Startup the Highlander IDE

2) Add the kbmMWSpider20.dll assembly to the references section of the project

3) Drop a GridView and Button control onto the Default.aspx page

4) Add the appropriate units to the implementation section of the page.

5) Implement the code for the button.

6) Run the application!

  

When I first ran the project up I hit a problem because I'd previously changed the application server return values.  However in debugging my ASP.net client the seemless debugging in Highlander was nicely highlighted.  Here is a screen shot of some C# code of kbmMWSpider that I traced into seemlessly.

 

There we have it.  kbmMWSpider working nicely in Highlander.

I'm planning another Blog showing the full client framework running but for now it's off to get some breakfast  

Richard

I've been building ASP.net web applications using kbmMW as the backend server since mid-2004.  All of the application servers have been Win32 based and of course the web frontend is dotnet based.  During this time I've used various techniques for transferring data between the tiers.

These include

• ASP.net built using Delphi for DotNet connecting to kbmMW using the Delphi based kbmMW clients exchanging data as VCL streams of objects
• ASP.net built using Delphi for DotNet connecting to kbmMW using the Delphi based kbmMW client using kbmMemtable DotNetAdditions (alpha) so that kbmMWQueries can be bound to directly
• ASP.net built using VS2005 and DotNet 2 using kbmMWSpider extensions for DotNet
• Something else I have codenamed Agility

What I'd like to share with you is the third of these because we will be releasing it very shortly after kbmMW 2.70.

It is a reality today that there is no CodeGear offering for developing DotNet applications for anything above version 1.1 of the framework.  However ASP.net 2 has some compelling features that we wish to tap into and certainly before the release of Highlander.

The clear choice then is VS2005 and C# as the programming language and then the kbmMWSpider client to connect to and interact with the application server. There is a problem with this approach though and that is the lack of kbmMemtable support running under dotnet 2.  After some pondering the solution was kbmMWSpider extensions for dotnet.

This is in use at http://www.myc4d.com/, http://www.turbomiddleware.com/ and some other interesting projects including an existing customer for their new web work (they contracted us to bring this forward in the roadmap). 

What is kbmMWSpider extensions for DotNet (yes we struggled with a name for this). 

It is comprised of three things

• an enhanced Spider C# client that supports connection pooling, request retries and failover
• a complete VCL compatible streaming layer
• two data formatters that accept kbmBinaryFormat (kbmMemtable in other words) and kbmMWFormat datastreams and decode these into System.Data.Datatable instances

In more detail

Enhanced Spider C# client that supports connection pooling, request retries and failover

An issue was dicovered when developing the portal.  Should the application server connection be lost for some reason the dotnet socket layer still reports the socket as being connected.  This sounds odd but in the world of sockets the only way generally to know if you're connection to the endpoint is valid is to attempt to send some data and see what happens.  If the connected property of the socket returned true then in anycase this would only be accurate at the point of the call - the socket could still lose it's connection a microsecond later. In the Delphi client we have robust handling of request failures with logic to retry the request and then attempt to failover to other servers.  The first incarnation of the Spider C# client didn't provide this logic so we have added it.

This issue was discovered because of another new feature added - connection pooling.  We now provide managed and transparent connection pooling to boost responsiveness and performance in high request situations. 

Anyone that has used TWriter/TReader in the VCL will know the power it provides.  It is also an integral part of the way kbmMW works.  Without this the scope for bridging the Visual Studio and VCL worlds is limited.

This is where the VCL support was trying to take us to.  Being able to make a request to a kbmMW application server, have it stream the results of a query into the resultstream and be able to decode and bind to this in the dotnet code.

Here is the real live deployed code for "mylicences.aspx" used in myC4D.com

Notice the line this.gridLicences.DataSource = dt line.  This is native data binding to the result of the MyLicences call.

Now the MyLicenceses call

Here is a regular request to a kbmMW application server that saves it's query results onto the resultstream.  In the ASP.net code we instantiate a C4D.kbmMW.Data.DataAdapter and get it to decode the contents using FillDatatable.  From this point on we have our data in a dotnet friendly form that we can bind to and manipulate. It is efficient and complete.

I'm looking for more people to play with this before release so if you're interested let me know.

Richard

We have had a few enquiries about the where abouts of kbmMW 2.70 so I thought I would explain what has been occuring.

As you are all aware we recently announced a whole new business model for our products.  The aim of this was explained in detail in the announcements.  It is something we have been discussing for a very long time and put off over and over again.  We finally made the announcements and were overwhelmed by the support and goodwill shown to us by our users expressed in the newsgroups, by email and directly to us on MSN and Skype.

It was then down to business - deliver 2.70.  This involved:-

1. finalising the myc4d.com portal

2. organising the server resources

3. code completing 2.70

4. biting the bullet and finalising kbmProject to manage the huge number of build scripts we have due to supporting all the Delphi/Kylix and soon FPC build configurations.

We were doing well on all four targets until I hit a few time issues and bugs in the new kbmMWSpider for DotNet and managed to miss the May 1 target for the portal.  I'll blog seperately on how this was built but it's very cool.  2 and 4 were completed.

Now down to 3.

In our wisdom we decided to add a wonderful new feature to kbmMW - AJAX support right out of the server.  Like any idea such as this it is a great idea until it isn't!  Suffice to say Kim ended up badly burnt by his bits and bytes but soon emerged with the goodies - it's looking good but the demo is not quite there yet.  Where we are at right now is trying to decide whether to make a 2.70 release and cut out the demo of the Ajax support and move that to a 2.71 or see if it can be sneaked into the release in the next day or two.  As Kim said "I would feel best releasing with the demo because then all the bits have been demonstrated to actually work rather than theoretically work" - enough said.

Suffice to say this delay is no fault of any of our customers so those that are on a 1 month SAU will be guaranteed access to the 2.70 release.

We will post a link to this very cool demo soon but to preview what we are doing here is the original version. What we will be showing very shortly is the same IM application running using all the same html, js, css, swf, graphics etc - in fact 100% the same content as the original but it will be communicating directly to and solely with a kbmMW server - it's impressive. For those that have worked with AJAX before you'll be interested to know this will also demonstrate how to use JSON directly out of Delphi

 

Richard

 

In quite a departure for us at Components4Developers we have decided to start blogging.  Day to day we constantly talk amongst ourselves about new ideas, techniques and ways to improve our products.  Some of these ideas get into the released code line, some get put into our research basket for a revisit and some get rejected.  Our customers though don’t see this process occurring, well that’s not quite true.  Customers that engage us in consultancy services get to see this first hand as often we are solving big issues for them through elegant additions to our products.

For example, the first internal implementation of the messaging in kbmMW was UDP based.  Why UDP?  - Because a customer commissioned us to extend the middleware to support publish/subscribe distribution of real-time video.

Another customer needed connectivity for PHP, C, C# etc from which came the Spider clients collection.

It is worth mentioning then that if you really need a feature added to any of our products you could always commission us to develop that for you on a shared IP basis.  If we think the feature will be of general benefit to our products then we will include it in the production source and maintain it through its life.  It never hurts to ask

Getting back to the blogs.  Here from time to time we will share with you some of the ideas we have in the works, or things we have noticed across the industry.  Or, we may just have a joke or a moan.

Everyone is invited to join in. We invite our C4D friends who wish to join our bloggers list to let us know and we'll setup an account for you. Those that would simply like to comment on, or syndicate our missives please go ahead.

On another note.  Theses blogs are powered by BlogEngine.net.  Check it out.  It is Open Source and based on ASP.net.  Maybe I'll find a reason to hook it up to a kbmMW application server sometime.

Richard and Kim